What's new in 0.15.21 --------------------- - Bugfix: creating a file copy on a storage pool dirve could fail if: 1) parallel copying is enabled; 2) one of the file copy would be created on the same drive as the landing zone; 3) another file copy was created on another storage pool drive; 4) the file copy on another drive was linked to (on the landing zone) BEFORE the same-drive file copy was created. - Fixes #324 - Bugfix: when creating a parent folder (eg. missing share folder in storage pool), we need to use the proper 'original folder' to copy the ownership + permission from. - Fixes #319 - Bugfix: don't show a warning about logrotate not being installed, when in fact it's man that isn't installed - Fixes #313 - Fixed build_vfs.sh for Samba 4.15 on Fedore; less verbose build_vfs.sh output - Fixes #316 - Balance task is now better at finding the right files to move first, for faster results What's new in 0.15.20 --------------------- - New: VFS module for 4.18 - Bugfix: deb-specific modules, starting at 4.17 : using non-bundled (i.e. shared) libraries (for ldb, tdb, tevent & talloc) - Ref: #312 - Bugfix: missing symlinks for libsmbd-base-samba4.so & libsamba-debug-samba4.so (on Debian 12 at least) - Ref: #312 What's new in 0.15.19 --------------------- - New: VFS module for Samba 4.17: small change in `openat_fn` function definition (const struct vfs_open_how *how param replaces int flags, mode_t mode) Fixes #311 - Improvement: Small improvements to --balance - Improvement: Migration full path fo utf8mb4 to support emoji in file name - Bugfix: not all MySQL version support all +/-HH:MM timezones; log an error when we try and fail to change the MySQL TZ (for the current session) Fixes #303 - Update install-greyhole-package.sh - Using deb [signed-by=...] instead of apt-key (deprecated) What's new in 0.15.18 --------------------- - Compiled VFS module for Samba 4.16, and included those in .rpm and .deb (thanks zefie) - Fix for compiling on Alpine 3.14 (Samba 4.14) - Bugfix: In a Docker container, 'greyhole --create-mem-spool' complained that "There was a problem creating the in-memory spool folder", even if that was not true What's new in 0.15.17 --------------------- - Bugfix: VFS module for Samba 4.13 should be compiled with the most recent available minor version of Samba; building using 4.13.0 creates a dependency on a shared library that is not available without compiling (libsmbd-conn-samba4.so); also recompile VFS for 4.15, 4.14 & 4.12 with the latest minor versions of those too, just in case. Fixes #298 - Fix for build_vfs with Samba 4.11; also applied fix from 021a4cbd on VFS module for Samba 4.11 What's new in 0.15.16 --------------------- - Bugfix: open and mkdir tasks logged by VFS modules were not including the full path to the file or folder, causing empty directories being created in the storage pool drives' shares' root, instead of in the proper subfolder eg. `mkdir /mnt/samba/ShareName/Folder/SubFolder` resulted in empty folders created as `/path/to/storage/pool/drive*/ShareName/SubFolder` - Bugfix: VFS module for Samba 4.15 only logged basename (instead of full path) for rename (mv), (sym)link, delete (unlink) and rmdir Fixes #297 - Improved build_vfs script; fixed typo in comment What's new in 0.15.15 --------------------- - Better handling of locked files/share in a frozen directory - Optimization (and logging at PERF level) in code that handles close tasks logged by VFS module Ref: #293 What's new in 0.15.14 --------------------- - Fix for PHP 8.1 What's new in 0.15.13 --------------------- - Bugfix: modifying a file could result in a broken symlink in the LZ - Bugfix: build_vfs might be missing perl-CPAN on yum-based OS - Bugfix: log about running build_vfs should mention sudo, since it's required - Typo in build_vfs log when installing com_err & heimdal-devel - Don't use "--with-system-heimdalkrb5" when building the Samba module on Alpine Linux (Docker image) - Fixed build_vfs for Alpine Linux 3.15 (Samba 4.15) - Improved log (stdout) for build_vfs script What's new in 0.15.12 --------------------- - New: Added VFS module for Samba 4.15.x - Improvement: Create scorecards-analysis.yml - Improvement: Scorecard suggestion: added SECURITY.md - Improvement: Scorecard suggestion: pin Docker base images for VFS builds What's new in 0.15.11 --------------------- - Improvement: WebUI: easier to see which toggle/tab is selected - Bugfix: When MySQL server uses a different TZ than PHP, change the MySQL TZ to match - Fixes #288 What's new in 0.15.10 --------------------- - New: `greyhole --cp` (alias `cpgh`) command: Copy a file or folder onto your storage pool without going through Samba - New: When building a new version, create Docker images for all platforms (x86_64 & armv6/v7/v8) - Improvement: Comment out sticky-files suggestions in example config file - Improvement: build_vfs script: when compiling Samba fail, tail (100 lines) the build log - Improvement: tail of build log might not be sufficient to see the actual error; will cat the log instead - Improvement: build_vfs script: tail of build log might not be sufficient to see the actual error; will cat the log instead; waiting 15s after ./configure starts before starting to show progress - Improvement: Improved caching of PHP loaded modules, when using greyhole-php wrapper script - Improvement: Small code improvements here and there - Improvement: Balancing: try using 100MB+ files first; if none are found, look for 10MB+ files, then 5MB+, 1MB+ and then any files. Allows balancing a share/storage pool drive that only contains small files. What's new in 0.15.8 -------------------- - WebUI: crashfix in DataTables AJAX call, when search field is empty - Bugfix: Skip trying to process files with emojis in their name Fixes #272 - Bugfix: Samba VFS module for Samba 4.13+ would sometimes log "open for writing" operation incorrectly; that would leave a bunch of spam in the tasks table in MySQL - New VFS module for Samba 4.14; now using Docker to build VFS modules (was using VirtualBox VMs & homelab server before) What's new in 0.15.7 -------------------- - New VFS module for Samba 4.14 (untested) What's new in 0.15.6 -------------------- - Crashfix: can't call gh_error_handler() when using PHP 8 Ref: #265 - WebUI: Bugfix: Storage Pool graph was not showing the correct amount of used space - Build VFS script: allow usage of CPAN when Perl module is installed, but cpan executable is not in $PATH What's new in 0.15.5 -------------------- - Bugfix: greyhole-dfree might fail, if it is executed by two different (Samba) users with incompatible umask: first user creates this file by connecting to any Greyhole share, and umask makes this file unwritable by the second user, who will then see 0 bytes free on the shares What's new in 0.15.4 -------------------- - Improvement: Samba module: support for compiler before C99 Ref: #259 - WebUI Bugfix: Typo: Connexion vs Connection Ref: #261 - WebUI Bugfix: Saving donation email was failing Ref: #261 What's new in 0.15.3 -------------------- - Bugfix: default for 'daemon_niceness' config was not properly applied, preventing the service from starting Fixes #258 - Bugfix: files open read-only (lsof' FD = ##r) are not locked - WebUI Bugfix: wrong tab was selected by default (Balance instead of Logs), in Status page - WebUI Bugfix: Recent log entries was always empty - WebUI Bugfix: some PHP warnings Ref: #256 What's new in 0.15.2 -------------------- - New 'email' hook, to receive notifications that are normally sent by email using another channel (Slack, PushBullet, etc.) Fixes #250 - WebUI: Bugfix: Make DataTables responsive Ref: https://datatables.net/forums/discussion/comment/92587/#Comment_92587 - WebUI: colored lines in Status > Past Tasks table (green for writes, red for deletes, yellow for renames, blue for maintenance) - WebUI: Status page 'current action' is now loaded using AJAX, at the same time the log is loaded - DevOps: yum repo is now updated using a docker container; host OS changed (to Ubuntu focal), and there is no createrepo package/binary available for that platform What's new in 0.15.1 -------------------- - New: Samba VFS module for Samba 4.13 Fixes #247 - Improvement: use `greyhole --config` instead of grep-ing the greyhole.conf file manually, in init scripts - Bugfix: fsck for all shares: email report was listing all the shares multiple times - WebUI: Added note about 30-days trial period for the Admin web UI to the LICENSE file - WebUI: New: show Queue stats in Status page (equivalent of greyhole --view-queue) - WebUI: New: Last fsck Report tab in Greyhole Admin | Status (which uses tabs now to display the various sections) - WebUI: New: Status | Past Tasks: browse past (already processes) tasks, with pagination & search - WebUI: New: start (and stop) fsck from new Actions page - WebUI: New: Start (and stop) balance from Actions page - WebUI: New: Emptying Trash from Actions page - WebUI: New: Pause/Resume in Actions page; also show Paused status in Status page, and --status output - WebUI: New: toggle to follow (tail) log in Status page; will refresh the "Recent log entries" every 10 secs (on by default) - WebUI: New: remove a drive from the storage pool - WebUI: New Trash Manager page Fixes #245 - WebUI: Improvement: Added note about mounting shares locally, at the end of the initial setup wizard - WebUI: Improvement: Change dark mode theme - WebUI: Improvement: Better detect what is the currently ongoing task - WebUI: Improvement: load all data using AJAX, and reload when changing tabs, to ensure all presented data is up to date (in Status and Storage pool tabs - others are static config/actions) - WebUI: Bugfix: don't show unfinished (written, but not closed) task, in Status page What's new in 0.15.0 -------------------- - New: Admin Web UI to monitor / control Greyhole (and Samba) - New: Web wizard for initial setup - Improvement: better log, when a file is kept open by another process - Improvement: when a file is modified, don't wake all storage pool drives unless necessary - Improvement: Less log spam when postponing a task for a frozen dir - Document: 'check_storage_pool_schedule' config - Removed: deprecated db_engine from example .conf and man page - Bugfix: use utf8mb4 charset for MySQL session; fixes issues with non-ASCII characters in du_stats - Bugfix: Stop loading Gone-FSCKed-Drives setting from DB over and over for no reason. Same for Gone-OK-Drives. Fixes: #244 - Bugfix: status table was not logging executed tasks, because the action column definition was wrong - Bugfix: get_debug_bt() was modifying the actual function arguments (objects and array), replacing them with string representations! - Bugfix: don't insert into settings table if it already contains something (an empty array, for example) - Bugfix: allow .deb to be installed in a Ubuntu Docker container, where modprobe is not installed, and /sys/modules/ are read-only - Bugfix: quiet PHP warning about file_put_contents() when trying to save a metafile eg. when drive is full; but log a warning What's new in 0.14.0 -------------------- - Improvement: Renamed `--replace` to `--replaced` - Improvement: Renamed `--going` and `--gone` to `--remove` Will prompt the user to know if we should (or not) try to copy files from the specified drive. Fixes #88 - Improvement: When executing --remove, check the copied files MD5 Fixes #242 - Improvement: `--remove` is now executed asynchronously (using the daemon) An email report is sent once it's done. Ref: #58 - Improvement: `--remove-share` is now executed asynchronously (using the daemon) An email report is sent once it's done. Fixes #58 - Bugfix: status table should be MYISAM, not InnoDB Fixes #241 - Bugfix: quiet PHP warning about file_put_contents() when trying to save a metafile eg. when drive is full - Bugfix: --balance-status triggered division-by-zero for drive groups that were empty What's new in 0.13.6 -------------------- - arm64: install the Samba VFS modules in the correct directory - Build script: make sure code is up to date with remote, before creating packages! What's new in 0.13.5 -------------------- - Bugfix: there was a 50% chance that Samba spooled tasks with the same modified-time timestamp would be incorrectly sorted, resulting in close-write-open ordering, instead of open-write-close. This resulted in a stuck daemon. Fixes #238 - Bugfix: Missing 4.12 support in .deb postinst script - Bugfix: when a CRITICAL error occurs during the initialize phase (eg. during config file parsing), don't log, but DO exit(1) Fixes #239 - New: Support for arm64 (aarch64) libdir; added .deb package for arm64 - Improvement: Simplified spooled tasks sorting code - Improvement: Added some steps and notes in INSTALL file, to help users manually install Greyhole What's new in 0.13.4 -------------------- - Bugfix: round free space number, in output of dfree Fixes #235 What's new in 0.13.3 -------------------- - greyhole-dfree was broken; missing required classes Fixes #235 What's new in 0.13.2 -------------------- - Crashfix: renaming (moving) a folder that was ignored Fixes #234 What's new in 0.13.1 -------------------- - Small fix for MySQL 8 Fixes #232 - Prevent crash when trying to send a inexistant fsck log file by email - Bugfix: free space calculation was not working, in docker, if multiple mounts (-v) were defined for a storage pool drive, and 1+ folders inside it What's new in 0.13.0 -------------------- New features - New command: greyhole --mv (beta; requires more testing) - Allows you to move data from one Greyhole share to another without copying the data through the Samba shares - Detect and fix bitrot/silent errors; requires a new cron job; see near the end of USAGE file Fixes #199 - New command: greyhole --print-fsck : Print the fsck report for the last completed fsck task. This will print the same content that is sent by email when the --email-report option is used. - Refactored FSCK report code; new layout for the 'all shares' fsck that runs weekly Improvements - greyhole-dfree (used by Samba to calculate free space) should take into account the num_copies option of the share; i.e. real free space = 1 GB, num_copies = 4 => available free space on Samba share = 250 MB; also added -h parameter to greyhole-dfree executable Fixes #156 - Save recent log entries in DB; use that in greyhole --status Fixes #114; Fixes #102 - Allow restarting Samba & Greyhole when running in Docker - Mention allow insecure wide links = yes as an alternative to unix extensions = no in USAGE file Ref: #192 Bugfixes - Correctly attribute write/close tasks; using the file handle (fd) was not enough; using the full path now - When a write task is logged, and no opened-for-writing task is in the DB, create it! Fixes issue where Greyhole would think a file is closed without being written to, when that is not the case. - When two spooled files (created by Samba VFS module) have the exact same modification time (to the microsecond!), ensure they are executed in the correct order: open, writes, close - When running --fsck --disk-usage-report, correctly update 'parents' du_stats, when deleting the checked folder's du_stats - --iostat was missing storage pool drives on un-numbered partitions (eg. whole-drive XFS partition on /dev/sdb); also added timestamps to output, updates are more frequent (5s instead of 10s), and sorted the storage pool drives in alphabetical order - Correctly balance storage pool drives, when drive_selection_groups are configured: will balance the drives in each group amongst themselves only, instead of trying to balance all the drives together. Fixes #198 - greyhole --process-spool logged warnings because Metastore backups config was uninitialized - If the Samba VFS symlink is missing/not working, and the expected target is also missing, download it from Github - greyhole --logs should not watch both .log and .err, as .log will always contains all the same logs as .err - Install script: ensure gnupg package is installed; it is required for apt-get add Fixes #227 - build_vfs script: try to install the required dependencies, before trying to build Ref: #223 - lsof file-locked check was not working in Docker (Alpine Linux), because -M flag is not available Various code improvement & refactoring - Removing old files; shellcheck; fixed white-space in shell scripts (was a mix of tabs and spaces) - Updated .patch to Samba 4.11 & 4.12 VFS module (was not working on Alpine 3.11 & Alpine Edge - 3.12) - WIP: VFS module for Samba 4.13 (untested) - Simplified manual INSTALL instructions about Samba VFS module; just need the correct binary in the LIB dir; daemon will take care of created the required symlink pointing there - Moved build-related files into build/ sub-directory - Create new Docker images during build - Faster re-building Samba VFS modules, on build servers What's new in 0.12.3 -------------------- - Process Samba spool ASAP, in order to prevent loosing the spooled tasks (stored in tmpfs) on reboot Fixes #224 What's new in 0.12.2 -------------------- - Bugfix: when a target storage drive file copy is on the same drive as the source file (most probably the LZ), skip this file when creating file copies in parallel; it will be copied afterwards using a rename Ref #225 - New config options: calculate_md5 and parallel_copying (default to yes) Ref: #225 - Updated instructions for latest Ubuntu - Better error logging when build_vfs.sh fails on ./configure - shellcheck for install script - Allow installation of old versions from deb repo Ref: #225 What's new in 0.12.1 -------------------- - Support for hard-links in Greyhole shares (only works on real files; i.e. *before* Greyhole moves the file to a storage pool drive) Fixed #220 - Added support for Samba 4.12 (untested) What's new in 0.12.0 -------------------- - Improvement: when multiple file copies are needed, create them in parallel using: cat source_file | tee target1 target2... Fixes #94 - Removed "Saving metadata in ..." log; too much spam, and serves no real purpose - Verify that 'testparm -s' returns something, else log a relevant critical error Fixes #221 What's new in 0.11.2 -------------------- - Build script: only install locally (on build server) if greyhole is already installed - Polyfill for is_iterable() function added in PHP 7.1 Ref #219 - Simpler weekly cron script What's new in 0.11.1 -------------------- - Bugfix: StoragePool::get_free_space() would sometimes return the wrong value (the free space on the last storage pool drive, instead of the specified drive) Fixes #217 What's new in 0.11.0 -------------------- - Build script: ensure .deb packages are created using GZip, (not XZ Utils) - DPKG repository doesn't work with *.xz files (yet) - When we want to restart the daemon, use supervisorctl, if Greyhole is running in a docker container using supervisord - Disk Usage web-app: remove prefix (current path) from chart - Added note about creating Samba users, and testing Samba shares before enabling Greyhole VFS - 'greyhole --logs' would output a warning in the error log (if one is defined in the config) was non-existent - Improvement: when executing a 'write' tasks, remove future 'write' tasks that are duplicates; i.e. don't execute multiple write tasks if only one would be enough - Removing php-mbstring dependency; it's already in the install script, and I added it to the INSTALL instructions Fixes #214 - New: log files checksums in DB, when new copies are created, or when checksums are checked (--fsck --checksums) - fsck checksums threads: re-start the ones that are missing; don't wait for ALL of them to disappear - Bugfix: When re-creating a file copy because its checksum is different, first make sure the storage pool drive has enough free space for this new file copy; if not, schedule to fsck_file task to create another file copy elsewhere - Pretty big code refactoring; some minor fixes too - Small code improvements; mostly fixing IDE warnings - Bugfix: don't try to manually escape values to inject into a SQL query! Use dynamic parameters instead. - Test cases script: small code improvements - Code change to support PHP 7.4 What's new in 0.10.24 -------------------- - Fixing patch file for samba 4.10 - Patch for compiling Samba on Alpine Linux - build_vfs.sh: correctly check which lib dir to put the greyhole.so symlink into - Support for Samba 4.11 What's new in 0.10.23 -------------------- - Bugfix: Detect custom (locally-compiled) Greyhole VFS module, and don't overwrite it What's new in 0.10.22 -------------------- - Update README.md - Merge pull request #209 from jult/patch-1 - Update install-greyhole-package.sh - Fixed some typos and missing commands - New check for missing shared libraries used by the Greyhole VFS module for Samba, and new script to use to fix those issues. - Updated script to install chkconfig and service if needed What's new in 0.10.21 -------------------- - Support for Samba 4.10 Fixes #207 What's new in 0.10.20 -------------------- - Bugfix: during file write operation (new or changed file), if a file copy is same as the source file (source_file_path = file_copy_file_path), no work is required! Just log something, and exit gh_copy_file() ASAP. - Bugfix: when file copy fails, make sure we don't leave a broken symlink in the landing zone - Bugfix: Only log 'Found a metadata file pointing to a drive not defined in your storage pool' if the metafile is not already marked as Gone. What's new in 0.10.19 -------------------- - Small code refactoring; don't loop on $metafiles if $share_file_link_to is FALSE - Bugfix: during balance, always try to create the symlink pointing to the newly moved file, even if there was no symlink before - Bugfix: during balance, load NOK metafiles too, while searching for the metafile which needs to be updated, after we moved a file - Refactored/simplified code that executes a fsck task - Improvement: Refactored how fsck for all shares are tracked, to be able to send a single email report once all the tasks have completed, instead of one email per share - Incorrect "Ended" time in fsck report for all shares What's new in 0.10.18 -------------------- - Bugfix: when a table is crashed, greyhole would go into an infinite loop trying to repair it Ref: #203 - Improvement to how --balance works (should be faster now) & improved the debug logs it creates - Bugfix: --balance-status didn't correctly display rows for storage pool with less free space than the minimum specified in greyhole.conf - Unused var What's new in 0.10.17 -------------------- What's new in 0.10.16 -------------------- - Improvement: check that the required PHP extensions are enabled, before executing (anything) - Fix warning about permissions, when trying to overwrite /tmp/greyhole-php.loaded.mods - Added note about keeping [global] above other section in smb.conf - Bugfix: Samba VFS module failed to log write operations that occurred asynchronously, triggering VFS' pwrite_send & pwrite_recv instead of pwrite. This resulted in Greyhole not noticing any file write ops, between the file open & close ops. - New: VFS modules for Samba 4.8 & 4.9 - Bugfix: folder where we need to put the VFS module symlink might not exist What's new in 0.10.15 -------------------- 912a913 - Bugfix: Fix mkdir/rmdir issue on Samba 4.5 What's new in 0.10.14 -------------------- - Add systemd service into releases - Samba 4.7 Support - Bugfix: libsmbd-base-samba4.so not found on Debian 9 & Ubuntu 17 (fixed for real this time, hopefully!) - Bugfix: --balance-status was not displaying properly for drives with 1TB+ of free space What's new in 0.10.13 --------------------- - Bugfix: VFS module was creating randomly-named folders in shares root. What's new in 0.10.12 --------------------- - Bugfix: libsmbd-base-samba4.so not found on Debian 9 & Ubuntu 17 What's new in 0.10.11 -------------------- - Bugfix: Endless loop when create_copies_from_metafiles() returns FALSE, when trying to process a 'write' op What's new in 0.10.10 --------------------- - Updated VFS module for Samba 4.5 (x86_64); old one was not working (on Debian Stretch at least) Error was: Error loading module '/usr/lib/x86_64-linux-gnu/samba/vfs/greyhole.so': /home/gb/samba-4.5.0/bin/shared/private/libsmbd-base-samba4.so: undefined symbol: s3_tevent_context_init - Bugfix: disk usage web-app: error when selected folder contains only files (no folders) - Bugfix: Don't add a she-bang in web-app/index.php during build - Bugfix: FATAL (crash) in Hook code when PHP warning was logged What's new in 0.10.9 -------------------- - Improved USAGE instructions that will skip trying to re-create the greyhole MySQL DB if it already exists - New `check_storage_pool_schedule` (undocumented) config option What's new in 0.10.8 -------------------- - Simplified .spec (RPM specs) file - Moved install script into git repo - Install the php-mysql(nd) package from the install script instead of using RPM requirement - Typo in .spec file What's new in 0.10.7 -------------------- - Future-proofing .rpm package (added new option for MySQL/MariaDB server package) - Bugfix: --balance-status was sometimes, depending on the free/used values, don't display properly, and would show PHP warnings - Samba 4.6 support What's new in 0.10.6 -------------------- - Removed reference to SQLite (not supported anymore) - Missing DB migration 12 when migration 11 ran in the past - Bugfix: Stop using open_basedir directive, which is causing weird issues with random users What's new in 0.10.5 -------------------- - Updated PHPStorm logo - Ensure complete columns in tasks/tasks_complete have the right type. What's new in 0.10.4 -------------------- - Bugfix: PDO error is sometimes in handle, sometimes in statement, always in PDOException What's new in 0.10.3 -------------------- - Bugfix: when trying to work on a file whose full_path is > 255 characters An error was thrown; will now try to enlarge the full_path columns in the various tables, to allow users who hit this limit get the larger columns they need. What's new in 0.10.2 -------------------- - Bugfix: complete = 'written' was not an option What's new in 0.10.1 -------------------- - TINYTEXT > VARCHAR(255) - More explicit error message when no file copies are found What's new in 0.10.0 -------------------- - Hooks! https://github.com/gboudreau/Greyhole/issues/10 - Improvement: log warnings (and +) to both .err and .log files - README: Added note about MTA required to send emails What's new in 0.9.68 -------------------- - New Samba 4.5 support - Bugfix: fix for missing symlinks on Ubuntu didn't work for 332-bit systems Those installs use /usr/lib/i386-linux-gnu as their libs folder - Bugfix: if fsck report can't be loaded from disk, just rename the file, and initialize a new empty fsck report. - Better error reports for files with wrong md5