fundamentals ∞
Some fundamental tools required by most self-respecting python-packages bundled in one place. Very opinionated project setup tools including logging, plain-text settings files and database connections.
Documentation for fundamentals is hosted by Read the Docs (development version and master version). The code lives on github. Please report any issues you find here.
Features ∞
How to cite fundamentals ∞
If you use fundamentals
in your work, please cite using the following BibTeX entry:
@software{Young_fundamentals,
author = {Young, David R.},
doi = {10.5281/zenodo.8037510},
license = {GPL-3.0-only},
title = ,
url = {https://github.com/thespacedoctor/fundamentals}
}
API Reference ∞
Modules
Common tools used thoughout the fundamentals package |
|
Tools used to perform download of files and HTML pages from the web |
|
Tools for working with files and folders |
|
Some handy mysql database query and insertion tools |
|
plugins for nose2 unit testing |
|
*Render python objects as various list and markup formats * |
|
Some reusable statistic functions |
|
Logger setup for python projects |
|
Some time functions to be used with logging etc |
Classes
A class to daemonise a python code |
|
The fileChunker iterator - iterate over large line-based files to reduce memory footprint |
|
rotating file handler for logging |
|
A fake logger object so user can set ``log=False`` if required |
|
a database object that can setup up a ssh tunnel (optional) and a database connection |
|
Take a sqlite database file and copy the tables within it to a MySQL database |
|
Take key-values from yaml files including a tablename(s) and add them to a mysql database table |
|
*The dataset object is a list of python dictionaries. |
|
common setup methods & attributes of the main function in cl-util |
|
Default setup for fundamentals style unit-testing workflow (all tests base on nose module) |
Functions
Get the root path for this python package |
|
append the current datestamp to the end of the filename (before the extension). |
|
get the filename from a URL. |
|
A datetime stamp to be appended to the end of filenames: ‘YYYYMMDDtHHMMSS’ |
|
get multiple url documents and place them in specified download directory/directories |
|
Convert a python list of dictionaries to pretty csv output |
|
list directory contents recursively. |
|
Add tags and ratings to your macOS files and folders |
|
multiprocess pool |
|
Setup and return a console logger |
|
setup dryx style python logging |
|
convert dictionary to mysql table |
|
A function to run all mysql scripts in a given directory (in a modified date order, oldest first) and then act on the script files in accordance with the succcess or failure of their execution |
|
get database table column names |
|
|
insert list of dictionaries into database tables |
Given a mysql query, read the data from the database and return the results as a list of dictionaries (database rows) |
|
Start a database connection using settings in yaml file |
|
Probe a database to determine if a given table exists |
|
Execute a MySQL write command given a sql query |
|
given a sorted list of values, median sigma-clip values based on a window of values either side of each value (rolling window) and return the array mask |
|
Return the time difference between two dates as a string |
|
*convert date to a relative datetime (e.g. |
|
A datetime stamp in MySQL format: ‘YYYY-MM-DDTHH:MM:SS’ |
Release Notes ∞
v2.4.1 - May 24, 2023
FIXED small bug fixes to daemonise code
v2.4.0 - May 24, 2023
FEATURE added a class to help daemonise code
v2.3.12 - May 10, 2022
FIXED doc fixes
v2.3.11 - March 17, 2022
FIXED: deadlocked connections now attempt to reconnect.
v2.3.10 - March 7, 2022
ENHANCEMENT: can now turn off multiprocessing with the
turnOffMP
parameter offmultiprocessing
. Needed for full profiling of code.
v2.3.9 - November 8, 2021
FIXED: moved depreciated calls to yaml
load
tosafe_load
v2.3.8 - September 29, 2021
ENHANCEMENT: fundamentals is now on conda-forge
v2.3.7 - September 27, 2021
ENHANCEMENT: some speed improvements in multi-downloads
v2.3.6 - August 16, 2021
FIXED: no longer crashing for scripts where no settings file is passed in via CL arguments
FIXED: database credentials can now be passed to the command-line again
v2.3.5 - July 30, 2021
FEATURE: code bases using fundamentals can now include a ‘advanced_settings.yaml’ file at the root of the project which will be read before the user settings file. User settings trump settings in this ‘advanced_settings.yaml’ file. The purpose is to be able to have hidden/development settings.
v2.3.4 - March 16, 2021
ENHANCEMENT: added loop to reattempt timed-out queries (up to 60 times)
v2.3.2 - February 23, 2021
FIXED: Logger was being set from default settings file even if a custom settings file given from command line
v2.3.1 - January 6, 2021
FIXED: astropy import causing grief with other package installs. Move to with function instead of module level import.
v2.3.0 - January 1, 2021
FEATURE: added a stats subpackage with a
rolling_window_sigma_clip
function
v2.2.9 - December 3, 2020
FIXED: relative time reporting (python 2>3ism)
v2.2.8 - November 12, 2020
fixed: logging levels
v2.2.7 - November 10, 2020
fixed: mysql port connection issue (with MaxScale proxy)
v2.2.6 - November 9, 2020
fixed: syntax error
v2.2.5 - November 2, 2020
enhancement: adding colour to logs
enhancement: addition of port in database connection settings
fixed: replacing depreciated ‘is” syntax with ==
v2.2.4 - May 28, 2020
enhancement: allowing ‘~’ as home directory for filepaths in all settings file parameters - will be converted when initially read
v2.2.3 - May 26, 2020
fixed: delimiters catered for in sql scripts
v2.2.2 - May 25, 2020
refactor:
list_of_dictionaries
now returns bytes decoded into UTF-8 string when rendered to other mark-up flavour.refactor: moved module level numpy imports so that packages with fundamentals as a dependency do not have numpy as a needless dependency
v2.2.1 - May 13, 2020
fixed:
directory_script_runner
functiondatabaseName
parameter changed to be optional
v2.2.0 - May 13, 2020
feature: new
execute_mysql_script
function that allows execution of a sql script directly from filerefactor: added a dbConn to the directory script runner
v2.1.7 - May 4, 2020
fixed: inspect module depreciation warnings stopped
fixed: MySQL error messages printed to stdout (previously hidden)
v2.1.3 - April 17, 2020
enhancement: cleaned up docs theme and structure. More documentation to come.