Some fundamental tools required by most self-respecting python-packages bundled in one place. Very opinionated project setup tools including logging, plain-text settings files and database connections.

Documentation for fundamentals is hosted by Read the Docs (development version and master version). The code lives on github. Please report any issues you find here.

API Reference



Common tools used thoughout the fundamentals package

Tools used to perform download of files and HTML pages from the web


Tools for working with files and folders


Some handy mysql database query and insertion tools


plugins for nose2 unit testing


*Render python objects as various list and markup formats *


Some reusable statistic functions


Logger setup for python projects


Some time functions to be used with logging etc



The fileChunker iterator - iterate over large line-based files to reduce memory footprint


rotating file handler for logging


A fake logger object so user can set ``log=False`` if required


a database object that can setup up a ssh tunnel (optional) and a database connection


Take a sqlite database file and copy the tables within it to a MySQL database


Take key-values from yaml files including a tablename(s) and add them to a mysql database table


*The dataset object is a list of python dictionaries.

common setup methods & attributes of the main function in cl-util


Default setup for fundamentals style unit-testing workflow (all tests base on nose module)



Get the root path for this python package

append the current datestamp to the end of the filename (before the extension).

get the filename from a URL.

A datetime stamp to be appended to the end of filenames: ‘YYYYMMDDtHHMMSS’

get multiple url documents and place them in specified download directory/directories


Convert a python list of dictionaries to pretty csv output


list directory contents recursively.


Add tags and ratings to your macOS files and folders


multiprocess pool


Setup and return a console logger


setup dryx style python logging


convert dictionary to mysql table


A function to run all mysql scripts in a given directory (in a modified date order, oldest first) and then act on the script files in accordance with the succcess or failure of their execution


get database table column names


insert list of dictionaries into database tables


Given a mysql query, read the data from the database and return the results as a list of dictionaries (database rows)


Start a database connection using settings in yaml file


Probe a database to determine if a given table exists


Execute a MySQL write command given a sql query


given a sorted list of values, median sigma-clip values based on a window of values either side of each value (rolling window) and return the array mask


Return the time difference between two dates as a string


*convert date to a relative datetime (e.g.


A datetime stamp in MySQL format: ‘YYYY-MM-DDTHH:MM:SS’

Release Notes

v2.3.12 - May 10, 2022

  • FIXED doc fixes

v2.3.11 - March 17, 2022

  • FIXED: deadlocked connections now attempt to reconnect.

v2.3.10 - March 7, 2022

  • ENHANCEMENT: can now turn off multiprocessing with the turnOffMP parameter of fmultiprocessing. Needed for full profiling of code.

v2.3.9 - November 8, 2021

  • FIXED: moved depreciated calls to yaml load to safe_load

v2.3.8 - September 29, 2021

  • ENHANCEMENT: fundamentals is now on conda-forge

v2.3.7 - September 27, 2021

  • ENHANCEMENT: some speed improvements in multi-downloads

v2.3.6 - August 16, 2021

  • FIXED: no longer crashing for scripts where no settings file is passed in via CL arguments

  • FIXED: database credentials can now be passed to the command-line again

v2.3.5 - July 30, 2021

  • FEATURE: code bases using fundamentals can now include a ‘advanced_settings.yaml’ file at the root of the project which will be read before the user settings file. User settings trump settings in this ‘advanced_settings.yaml’ file. The purpose is to be able to have hidden/development settings.

v2.3.4 - March 16, 2021

  • ENHANCEMENT: added loop to reattempt timed-out queries (up to 60 times)

v2.3.2 - February 23, 2021

  • FIXED: Logger was being set from default settings file even if a custom settings file given from command line

v2.3.1 - January 6, 2021

  • FIXED: astropy import causing grief with other package installs. Move to with function instead of module level import.

v2.3.0 - January 1, 2021

  • FEATURE: added a stats subpackage with a rolling_window_sigma_clip function

v2.2.9 - December 3, 2020

  • FIXED: relative time reporting (python 2>3ism)

v2.2.8 - November 12, 2020

  • fixed: logging levels

v2.2.7 - November 10, 2020

  • fixed: mysql port connection issue (with MaxScale proxy)

v2.2.6 - November 9, 2020

  • fixed: syntax error

v2.2.5 - November 2, 2020

  • enhancement: adding colour to logs

  • enhancement: addition of port in database connection settings

  • fixed: replacing depreciated ‘is” syntax with ==

v2.2.4 - May 28, 2020

  • enhancement: allowing ‘~’ as home directory for filepaths in all settings file parameters - will be converted when initially read

v2.2.3 - May 26, 2020

  • fixed: delimiters catered for in sql scripts

v2.2.2 - May 25, 2020

  • refactor:list_of_dictionaries now returns bytes decoded into UTF-8 string when rendered to other mark-up flavour.

  • refactor: moved module level numpy imports so that packages with fundamentals as a dependency do not have numpy as a needless dependency

v2.2.1 - May 13, 2020

  • fixed:directory_script_runner function databaseName parameter changed to be optional

v2.2.0 - May 13, 2020

  • feature: new execute_mysql_script function that allows execution of a sql script directly from file

  • refactor: added a dbConn to the directory script runner

v2.1.7 - May 4, 2020

  • fixed: inspect module depreciation warnings stopped

  • fixed: MySQL error messages printed to stdout (previously hidden)

v2.1.3 - April 17, 2020

  • enhancement: cleaned up docs theme and structure. More documentation to come.