fundamentals

https://zenodo.org/badge/52598459.svg

https://img.shields.io/pypi/pyversions/fundamentals https://img.shields.io/pypi/v/fundamentals https://img.shields.io/conda/vn/conda-forge/fundamentals.svg https://pepy.tech/badge/fundamentals https://img.shields.io/github/license/thespacedoctor/fundamentals

http://157.245.42.153:8080/buildStatus/icon?job=fundamentals%2Fmaster&subject=build%20master http://157.245.42.153:8080/buildStatus/icon?job=fundamentals%2Fdevelop&subject=build%20dev https://cdn.jsdelivr.net/gh/thespacedoctor/fundamentals@master/coverage.svg https://readthedocs.org/projects/fundamentals/badge/?version=master https://img.shields.io/github/issues/thespacedoctor/fundamentals/type:%20bug?label=bug%20issues

Some fundamental tools required by most self-respecting python-packages bundled in one place. Very opinionated project setup tools including logging, plain-text settings files and database connections.

Documentation for fundamentals is hosted by Read the Docs (development version and master version). The code lives on github. Please report any issues you find here.

Features

How to cite fundamentals

If you use fundamentals in your work, please cite using the following BibTeX entry:

@software{Young_fundamentals,
    author = {Young, David R.},
    doi = {10.5281/zenodo.8037510},
    license = {GPL-3.0-only},
    title = ,
    url = {https://github.com/thespacedoctor/fundamentals}
}

API Reference

Modules

fundamentals.commonutils

Common tools used thoughout the fundamentals package

fundamentals.download

Tools used to perform download of files and HTML pages from the web

fundamentals.files

Tools for working with files and folders

fundamentals.mysql

Some handy mysql database query and insertion tools

fundamentals.nose2_plugins

plugins for nose2 unit testing

fundamentals.renderer

*Render python objects as various list and markup formats *

fundamentals.stats

Some reusable statistic functions

fundamentals.logs

Logger setup for python projects

fundamentals.times

Some time functions to be used with logging etc

Classes

fundamentals.daemonise

A class to daemonise a python code

fundamentals.files.fileChunker

The fileChunker iterator - iterate over large line-based files to reduce memory footprint

fundamentals.logs.GroupWriteRotatingFileHandler

rotating file handler for logging

fundamentals.logs.emptyLogger

A fake logger object so user can set ``log=False`` if required

fundamentals.mysql.database

a database object that can setup up a ssh tunnel (optional) and a database connection

fundamentals.mysql.sqlite2mysql

Take a sqlite database file and copy the tables within it to a MySQL database

fundamentals.mysql.yaml_to_database

Take key-values from yaml files including a tablename(s) and add them to a mysql database table

fundamentals.renderer.list_of_dictionaries

*The dataset object is a list of python dictionaries.

fundamentals.tools

common setup methods & attributes of the main function in cl-util

fundamentals.utKit

Default setup for fundamentals style unit-testing workflow (all tests base on nose module)

Functions

fundamentals.commonutils.getpackagepath

Get the root path for this python package

fundamentals.download.append_now_datestamp_to_filename

append the current datestamp to the end of the filename (before the extension).

fundamentals.download.extract_filename_from_url

get the filename from a URL.

fundamentals.download.get_now_datetime_filestamp

A datetime stamp to be appended to the end of filenames: ‘YYYYMMDDtHHMMSS’

fundamentals.download.multiobject_download

get multiple url documents and place them in specified download directory/directories

fundamentals.files.list_of_dictionaries_to_mysql_inserts

Convert a python list of dictionaries to pretty csv output

fundamentals.files.recursive_directory_listing

list directory contents recursively.

fundamentals.files.tag

Add tags and ratings to your macOS files and folders

fundamentals.fmultiprocess

multiprocess pool

fundamentals.logs.console_logger

Setup and return a console logger

fundamentals.logs.setup_dryx_logging

setup dryx style python logging

fundamentals.mysql.convert_dictionary_to_mysql_table

convert dictionary to mysql table

fundamentals.mysql.directory_script_runner

A function to run all mysql scripts in a given directory (in a modified date order, oldest first) and then act on the script files in accordance with the succcess or failure of their execution

fundamentals.mysql.get_database_table_column_names

get database table column names

fundamentals.mysql.insert_list_of_dictionaries_into_database_tables

insert list of dictionaries into database tables

fundamentals.mysql.readquery

Given a mysql query, read the data from the database and return the results as a list of dictionaries (database rows)

fundamentals.mysql.setup_database_connection

Start a database connection using settings in yaml file

fundamentals.mysql.table_exists

Probe a database to determine if a given table exists

fundamentals.mysql.writequery

Execute a MySQL write command given a sql query

fundamentals.stats.rolling_window_sigma_clip

given a sorted list of values, median sigma-clip values based on a window of values either side of each value (rolling window) and return the array mask

fundamentals.times.calculate_time_difference

Return the time difference between two dates as a string

fundamentals.times.datetime_relative_to_now

*convert date to a relative datetime (e.g.

fundamentals.times.get_now_sql_datetime

A datetime stamp in MySQL format: ‘YYYY-MM-DDTHH:MM:SS’

Release Notes

v2.4.1 - May 24, 2023

  • FIXED small bug fixes to daemonise code

v2.4.0 - May 24, 2023

  • FEATURE added a class to help daemonise code

v2.3.12 - May 10, 2022

  • FIXED doc fixes

v2.3.11 - March 17, 2022

  • FIXED: deadlocked connections now attempt to reconnect.

v2.3.10 - March 7, 2022

  • ENHANCEMENT: can now turn off multiprocessing with the turnOffMP parameter of fmultiprocessing. Needed for full profiling of code.

v2.3.9 - November 8, 2021

  • FIXED: moved depreciated calls to yaml load to safe_load

v2.3.8 - September 29, 2021

  • ENHANCEMENT: fundamentals is now on conda-forge

v2.3.7 - September 27, 2021

  • ENHANCEMENT: some speed improvements in multi-downloads

v2.3.6 - August 16, 2021

  • FIXED: no longer crashing for scripts where no settings file is passed in via CL arguments

  • FIXED: database credentials can now be passed to the command-line again

v2.3.5 - July 30, 2021

  • FEATURE: code bases using fundamentals can now include a ‘advanced_settings.yaml’ file at the root of the project which will be read before the user settings file. User settings trump settings in this ‘advanced_settings.yaml’ file. The purpose is to be able to have hidden/development settings.

v2.3.4 - March 16, 2021

  • ENHANCEMENT: added loop to reattempt timed-out queries (up to 60 times)

v2.3.2 - February 23, 2021

  • FIXED: Logger was being set from default settings file even if a custom settings file given from command line

v2.3.1 - January 6, 2021

  • FIXED: astropy import causing grief with other package installs. Move to with function instead of module level import.

v2.3.0 - January 1, 2021

  • FEATURE: added a stats subpackage with a rolling_window_sigma_clip function

v2.2.9 - December 3, 2020

  • FIXED: relative time reporting (python 2>3ism)

v2.2.8 - November 12, 2020

  • fixed: logging levels

v2.2.7 - November 10, 2020

  • fixed: mysql port connection issue (with MaxScale proxy)

v2.2.6 - November 9, 2020

  • fixed: syntax error

v2.2.5 - November 2, 2020

  • enhancement: adding colour to logs

  • enhancement: addition of port in database connection settings

  • fixed: replacing depreciated ‘is” syntax with ==

v2.2.4 - May 28, 2020

  • enhancement: allowing ‘~’ as home directory for filepaths in all settings file parameters - will be converted when initially read

v2.2.3 - May 26, 2020

  • fixed: delimiters catered for in sql scripts

v2.2.2 - May 25, 2020

  • refactor:list_of_dictionaries now returns bytes decoded into UTF-8 string when rendered to other mark-up flavour.

  • refactor: moved module level numpy imports so that packages with fundamentals as a dependency do not have numpy as a needless dependency

v2.2.1 - May 13, 2020

  • fixed:directory_script_runner function databaseName parameter changed to be optional

v2.2.0 - May 13, 2020

  • feature: new execute_mysql_script function that allows execution of a sql script directly from file

  • refactor: added a dbConn to the directory script runner

v2.1.7 - May 4, 2020

  • fixed: inspect module depreciation warnings stopped

  • fixed: MySQL error messages printed to stdout (previously hidden)

v2.1.3 - April 17, 2020

  • enhancement: cleaned up docs theme and structure. More documentation to come.