Add apt_update_safe() function that warns instead of aborting when apt-get update fails (e.g. enterprise repo 401 Unauthorized). Shows a helpful hint about disabling the enterprise repo when no subscription is active. Replaces direct apt-get update calls in build.func and install.func.
Prevent terminal job-control signals from suspending the script during recovery by trapping TSTP, TTIN and TTOU (instead of only TSTP) and restoring them on exit. Also clear the terminal 'tostop' flag in stop_spinner() with `stty -tostop` to avoid background spinner I/O from stopping the process group.
* fix(zammad): configure Elasticsearch for LXC container startup
- Set discovery.type: single-node (required for single-node ES)
- Set xpack.security.enabled: false (not needed in local LXC)
- Set bootstrap.memory_lock: false (fails in unprivileged LXC)
- Add startup wait loop (up to 60s) to ensure ES is ready before
Zammad installation continues
Fixes #12301-related recurring Elasticsearch startup failures
* refactor(api): eliminate duplicate traps, harden error handling & telemetry
Phase 1 - Structural:
- Remove api_exit_script() and 5 inline traps from build.func
- error_handler.func is now the sole trap owner via catch_errors()
- Update api.func comment reference (api_exit_script -> on_exit)
Phase 2 - Quality:
- Add stop_spinner() + cursor restore to error_handler(), on_interrupt(),
on_terminate(), on_hangup() to prevent spinner/cursor artifacts
- Enhance _send_abort_telemetry() with error text (last 20 log lines),
duration calculation, and 2 retry attempts (was fire-and-forget)
- Harden json_escape() to also strip DEL (0x7F) character
* fix(build): show spinner during post_update_to_api to prevent Ctrl+Z abort
post_update_to_api can take up to 33 seconds worst-case (3 curl attempts
x 10s timeout + sleep delays). Without any terminal output during this
time, users think the script is stuck and press Ctrl+Z, which prevents
the recovery menu from ever appearing.
Add msg_info spinner before both post_update_to_api calls in the failure
path (initial report + final force retry after recovery menu).
* fix(build): prevent SIGTSTP from killing recovery dialog
- Replace msg_info/stop_spinner with plain echo for telemetry reporting
The background spinner process in non-interactive shells (bash -c)
can trigger SIGTSTP, stopping the entire process group before the
recovery dialog appears. Plain echo avoids this.
- Add trap '' TSTP at failure path entry to ignore suspension signals
Prevents Ctrl+Z or terminal-related SIGTSTP from interrupting the
recovery menu. Restored with trap - TSTP before exit.
- Root cause: msg_info starts a background process (spinner &) that
is not properly detached in non-interactive shells where job control
(set -m) is OFF. The disown builtin has no effect without job
control, leaving the spinner in the same process group. This can
cause terminal I/O conflicts during the 33-second post_update_to_api
retry window, resulting in [2]+ Stopped.
* fix(test): initialize colors and remove illegal local in test harness
- Call load_functions() after sourcing core.func to initialize
color/formatting/icon variables (RD, GN, YW, CL, TAB, etc.)
- Remove 'local' keyword from top-level scope (not inside function)
- Default REPO_SOURCE to ref_api instead of main
* chore: remove test-recovery-dialog.sh from branch
* Revert "fix(zammad): configure Elasticsearch for LXC container startup"
This reverts commit 10e450b72f.
* fix(build): show telemetry status only in verbose mode
Telemetry reporting is an implementation detail that doesn't help
the user during failure recovery. Wrap echo statements with
VERBOSE check so they only appear when verbose mode is enabled.
Validation status was persisting through container start, network check,
and OS customization. Now transitions to 'configuring' immediately after
create_lxc_container returns. Validation only covers storage/template/cluster
checks as intended.
Pipe lxc-attach output through tee into /tmp/.install-capture-${SESSION_ID}.log and use PIPESTATUS[0] to get the real lxc-attach exit code. Prefer a pulled container-side INSTALL_LOG when it exists and is >100 bytes; otherwise fall back to the host-captured terminal log (stripping ANSI codes) and append it to the combined log so get_full_log() can find it. Apply the same capture behavior to the retry path and remove temporary capture files on completion. This makes install output reliable when container-side logging is missing (DNS errors, early crashes, or missing silent() usage).
After 'export INSTALL_LOG' in build.func, get_active_logfile() returned
the container's INSTALL_LOG path for all host-side logging, causing
msg_info/msg_ok/msg_error on the host to write to /root/.install-SESSION.log
(the host file, not the container's) instead of BUILD_LOG. This made
BUILD_LOG incomplete and get_full_log() unable to send full traces.
Fix: Add _HOST_LOGFILE (not exported, invisible to container) so the host
always logs to BUILD_LOG. Container still uses INSTALL_LOG as before.
* Enhance telemetry, signal handling, and logs
Improve failure telemetry and signal handling across the installer: add get_full_log() to collect/strip/truncate install logs and include them in API payloads with a truncated retry; add CONTAINER_INSTALLING flag around lxc-attach and stop containers on abort to avoid orphaned "installing/configuring" records; introduce _send_abort_telemetry() (curl fallback for container context) and _stop_container_if_installing() helpers; centralize and simplify EXIT/ERR/INT/TERM/HUP traps and handlers (including a new on_hangup handler) and update VM scripts to report numeric exit codes. Also ensure best-effort log collection is performed and tweak error categorization for certain signals.
* Include full log in error telemetry
Use get_full_log (up to 120KB) to populate the error telemetry field so the API receives the full installation trace; fall back to get_error_text (last ~20 lines) if the full log is empty. Removed collection and inclusion of a separate install_log field from the JSON payloads and simplified the retry payloads/comments accordingly. The change ensures error reports contain the complete trace while avoiding duplicate large log fields and keeps graceful failure handling (get_full_log || true).
* Anonymize IP addresses in get_full_log
Mask IPv4 addresses in logs when collecting full log output: added a sed step that replaces the last two octets with "x.x" to avoid exposing full IPs (GDPR). Also updated the comment to reflect anonymization; existing steps that strip carriage returns and ANSI escape sequences remain in place before truncating with head -c.
Prevent host-side error_handler from being triggered during in-container install/recovery by delaying re-enabling set -Eeuo pipefail and the ERR trap in misc/build.func until after install/recovery completes; add explanatory comments. Update misc/error_handler.func to fall back to BUILD_LOG if container-internal log path is unavailable, show the last 20 log lines when present, refine container vs host detection (check INSTALL_LOG file and /root), copy INSTALL_LOG into /root and write a .failed flag with the exit code for host-side detection, and ensure full-log output and container removal prompt are shown appropriately in host context. Tweak misc/core.func silent() output to include a "Full log" path and adjust formatting.
Introduce post_progress_to_api() in misc/api.func — a non-blocking, fire-and-forget curl ping (gated by DIAGNOSTICS and RANDOM_UUID) that updates telemetry status to "configuring". Wire this progress ping into multiple scripts (alpine-install.func, install.func, build.func, core.func) at key milestones (container start, network ready, customization, creation, cleanup) and replace/deduplicate some earlier post_to_api calls. Also update error_handler.func to always report failures immediately via post_update_to_api to ensure failures are captured even before/after container lifecycle.
Prevent hangs when pulling logs from containers by wrapping pct pull calls with timeout (8s) and running ensure_log_on_host under timeout (10s). Always send telemetry (post_update_to_api) before attempting best-effort log collection so status is reported even if log retrieval blocks. Update EXIT/ERR/SIGHUP/SIGINT/SIGTERM traps and consolidate error/interrupt handlers to use the new timeouted log collection. Changes in misc/build.func and misc/error_handler.func.
* fix: send telemetry BEFORE log collection in signal handlers
- Swap ensure_log_on_host/post_update_to_api order in on_interrupt, on_terminate, api_exit_script, and inline SIGHUP/SIGINT/SIGTERM traps
- For signal exits (>128): send telemetry immediately, then best-effort log collection
- Add 2>/dev/null || true to all I/O in signal handlers to prevent SIGPIPE
- Fix on_exit: exit_code=0 now reports 'done' instead of 'failed 1'
- Root cause: pct pull hangs on dying containers blocked telemetry updates, leaving 595+ records stuck in 'installing' daily
* feat: add execution_id to all telemetry payloads
- Generate EXECUTION_ID from RANDOM_UUID in variables()
- Export EXECUTION_ID to container environment
- Add execution_id field to all 8 API payloads in api.func
- Add execution_id to post_progress_to_api in install.func and alpine-install.func
- Fallback to RANDOM_UUID when EXECUTION_ID not set (backward compat)
* fix: correct telemetry type values for PVE and addon scripts
- PVE scripts (tools/pve/*): change type 'tool' -> 'pve'
- Addon scripts (tools/addon/*): fix 4 scripts that wrongly used 'tool' -> 'addon'
(netdata, add-tailscale-lxc, add-netbird-lxc, all-templates)
- api.func: post_tool_to_api sends type='pve', default fallback 'pve'
- Aligns with PocketBase categories: lxc, vm, pve, addon
* fix: persist diagnostics opt-in inside containers for addon telemetry
- install.func + alpine-install.func: create /usr/local/community-scripts/diagnostics
inside the container when DIAGNOSTICS=yes (from build.func export)
- Enables addon scripts running later inside containers to find the opt-in
- Update init_tool_telemetry default type from 'tool' to 'pve'
* refactor: clean up diagnostics/telemetry opt-in system
- diagnostics_check(): deduplicate heredoc (was 2x 22 lines), improve whiptail
text with clear what/what-not collected, add telemetry + privacy links
- diagnostics_menu(): better UX with current status, clear enable/disable
buttons, note about existing containers
- variables(): change DIAGNOSTICS default from 'yes' to 'no' (safe: no
telemetry before user consents via diagnostics_check)
- install.func + alpine-install.func: persist BOTH yes AND no in container
so opt-out is explicit (not just missing file = no)
- Fix typo 'menue' -> 'menu' in config file comments
* fix: no pre-selection in telemetry dialog, link to telemetry-service README
- Add --defaultno so 'No, opt out' is focused by default (user must Tab to Yes)
- Change privacy link from discussions/1836 to telemetry-service#privacy--compliance
* fix: use radiolist for telemetry dialog (no pre-selection)
- Replace --yesno with --radiolist: user must actively SPACE-select an option
- Both options start as OFF (no pre-selection)
- Cancel/Exit defaults to 'no' (opt-out)
* simplify: inline telemetry dialog text like other whiptail dialogs
* improve: telemetry dialog with more detail, link to PRIVACY.md
- Add what we collect / don't collect sections back to dialog
- Link to telemetry-service/docs/PRIVACY.md instead of README anchor
- Update config file comment with same link
* core: add progress; fix exit status
Introduce post_progress_to_api() in alpine-install.func and install.func to send a lightweight, fire-and-forget telemetry ping (HTTP POST) that updates an existing telemetry record to "configuring" when DIAGNOSTICS=yes and RANDOM_UUID is set. The function is non-blocking (curl -m 5, errors ignored) and is invoked during container setup and after OS updates to signal active progress. Also adjust api_exit_script() in build.func to report success (post_update_to_api "done" "0") for cases where the script exited normally but a completion status wasn't posted, instead of reporting failure.
* Safer tools.func load and improved error handling
Replace process-substitution sourcing of tools.func with an explicit curl -> variable -> source via /dev/stdin, adding failure messages and a check that expected functions (e.g. fetch_and_deploy_gh_release) are present (misc/alpine-install.func, misc/install.func). Add categorize_error mapping for exit code 10 -> "config" (misc/api.func). Tweak build.func: minor pipeline formatting and change the ERR trap to capture the actual exit code and only call ensure_log_on_host/post_update on non-zero exits, preventing erroneous failure reporting.
* tools: add data init and auto-reporting to tools and pve section
Introduce telemetry helpers in misc/api.func: _telemetry_report_exit (reports success/failure via post_tool_to_api/post_addon_to_api) and init_tool_telemetry (reads DIAGNOSTICS, starts install timer and installs an EXIT trap to auto-report). Integrate telemetry into many tools/addon and tools/pve scripts by sourcing the remote api.func and calling init_tool_telemetry (guarded with declare -f). Also apply a minor arithmetic formatting tweak in misc/build.func for RECOVERY_ATTEMPT.
* core: add progress; fix exit status
Introduce post_progress_to_api() in alpine-install.func and install.func to send a lightweight, fire-and-forget telemetry ping (HTTP POST) that updates an existing telemetry record to "configuring" when DIAGNOSTICS=yes and RANDOM_UUID is set. The function is non-blocking (curl -m 5, errors ignored) and is invoked during container setup and after OS updates to signal active progress. Also adjust api_exit_script() in build.func to report success (post_update_to_api "done" "0") for cases where the script exited normally but a completion status wasn't posted, instead of reporting failure.
* Safer tools.func load and improved error handling
Replace process-substitution sourcing of tools.func with an explicit curl -> variable -> source via /dev/stdin, adding failure messages and a check that expected functions (e.g. fetch_and_deploy_gh_release) are present (misc/alpine-install.func, misc/install.func). Add categorize_error mapping for exit code 10 -> "config" (misc/api.func). Tweak build.func: minor pipeline formatting and change the ERR trap to capture the actual exit code and only call ensure_log_on_host/post_update on non-zero exits, preventing erroneous failure reporting.
* Ensure API update is sent on script exit
Add exit-time telemetry handling across scripts to avoid orphaned "installing" records. Introduce local exit_code capture in api_exit_script and cleanup handlers and, when POST_TO_API_DONE is true but POST_UPDATE_DONE is not, post a final status (marking failures on non-zero exit codes, or marking done/failed in VM cleanups based on exit code). Changes touch misc/build.func, misc/vm-core.func and various vm/*-vm.sh cleanup functions to reliably send post_update_to_api on normal or early exits.
* Update api.func
* fix(telemetry): add missing exit codes to explain_exit_code()
- Add curl error codes: 4, 5, 8, 23, 25, 30, 56, 78
- Add code 10: Docker/privileged mode required (used in ~15 scripts)
- Add code 75: Temporary failure (retry later)
- Add BSD sysexits.h codes: 64-77
- Sync error_handler.func fallback with canonical api.func
- Add 'exit 130' after SIGINT trap handler
- Add 'exit 143' after SIGTERM trap handler
- Fixes issue where interrupted scripts stayed as 'Installing' in telemetry
- Previously the traps only sent the update but didn't terminate the script
- Query /cluster/resources via pvesh to check all VMs/CTs on ALL nodes
- Check /etc/pve/nodes/*/qemu-server and /etc/pve/nodes/*/lxc dirs
- Handles pmxcfs sync delays that caused sporadic ID conflicts
- Remove duplicate validate_container_id/get_valid_container_id definitions
- Add max_attempts safeguard to prevent infinite loops
* fix(telemetry): improve error reporting with structured error strings and better categorization
- Add build_error_string() that creates structured format:
'exit_code=N | description\n---\n<last 20 log lines>'
- Fix categorize_error() to map ALL known exit codes:
- Added: shell(1,2), proxmox(200-231), service(150-154),
database(170-193), runtime(243-249), signal(139,141,143)
- Split timeout from network (28 was in both)
- Added DPKG(255) to dependency category
- Update all API functions to use build_error_string():
post_update_to_api, post_update_to_api_extended,
post_tool_to_api, post_addon_to_api
- Add ensure_log_on_host() calls to on_exit, on_interrupt,
on_terminate handlers to prevent race condition where
telemetry reports before container log is pulled to host
* fix(ui): improve error output formatting and remove redundant log paths
- error_handler: Use msg_info/msg_ok/msg_warn for container cleanup
instead of raw echo with manual ANSI codes
- error_handler: Add ❓ icon before 'Remove broken container?' prompt
- error_handler: Indent log output with TAB for visual consistency
- build.func: Use msg_custom for installation log path display
- build.func: Use msg_info → msg_ok for container removal flow
- build.func: Use msg_warn for 'kept for debugging' message
- core.func/vm-core.func: Remove redundant container-internal log
path display (📋 View full log) since combined log on host is
the canonical location shown after failure
Enhance post_update_to_api to support a "force" mode and robust retry logic: add a 3rd-arg bypass to duplicate suppression, capture a short error summary, and perform up to three POST attempts (full payload, shortened error payload, minimal payload) with HTTP code checks and small backoffs. Mark POST_UPDATE_DONE on success (or after three attempts) to avoid infinite retries. Also invoke post_update_to_api with the "force" flag from cleanup paths in build.func and error_handler.func so a final status update is attempted after cleanup.
Copy the container install log to a host path before reporting a failure to the telemetry API so get_error_text() can read it. Introduce host_install_log and point INSTALL_LOG to the host copy when pulled via pct, move post_update_to_api after the log copy, and update the displayed installation-log path.
* Remove Go API and extend misc/api.func
Delete the Go-based API (api/main.go, api/go.mod, api/go.sum, api/.env.example) and significantly enhance misc/api.func. The shell telemetry file now includes telemetry configuration, repo source detection, GPU/CPU/RAM detection, expanded explain_exit_code mappings, and refactored post_to_api/post_to_api_vm to send non-blocking telemetry to telemetry.community-scripts.org while respecting DIAGNOSTICS/DEV_MODE and adding richer metadata (cpu/gpu/ram/repo_source). Also updates header/author info and improves privacy/robustness and error handling.
* Start install timer and refine error reporting
Call start_install_timer during build startup and overhaul exit/error reporting.
Changes:
- Invoke start_install_timer early in misc/build.func to track install duration.
- Update api_exit_script comments to reference PocketBase/api.func and adjust ERR/SIGINT/SIGTERM traps to post numeric exit codes (use $? / 130 / 143) instead of command strings.
- Replace the previous explain_exit_code implementation with a conditional fallback: only define explain_exit_code if not already provided (api.func is the canonical source). Expanded and reorganized exit code mappings (curl, timeout, systemd, Node/Python/Postgres/MySQL/MongoDB, Proxmox, etc.).
- In error_handler: stop echoing the container log path (host shows combined log), and post a "failed" update to the API with the exit code before offering container cleanup.
Rationale: these changes make telemetry more consistent and robust (numeric codes), provide a safe fallback for exit descriptions when api.func isn't loaded, and ensure failures are reported to the API prior to any automatic cleanup.
* Report install start/failure to telemetry API
Add telemetry hooks in misc/build.func: call post_to_api at installation start to capture early or immediately-failing installs, and call post_update_to_api with status "failed" and the install exit code when a container installation fails. This improves visibility into install failures for monitoring/telemetry.
* feat(build.func): update dynamic values in LXC profile on update_motd_ip
- Updates only OS/Hostname/IP lines in /etc/profile.d/00_lxc-details.sh
- Checks if values changed before updating (avoids unnecessary I/O)
- Preserves user customizations (app name, GitHub link, custom lines)
- Only updates if file exists and contains 'community-scripts' marker
- Fixes outdated OS version display after in-place upgrades (e.g., Bookworm → Trixie)
- Now reads OS name/version from /etc/os-release at login time
Fixes community-scripts/ProxmoxVE issue with static MOTD after OS upgrade
* add update_motd_ip in routine
* fix(core): add IPv6 fallback support to get_current_ip functions
Fixes IP detection in IPv6-only environments by adding fallback:
- Try IPv4 first (existing behavior)
- Fall back to IPv6 via interface lookup (eth0 scope global)
- Fall back to IPv6 routing with Google/Cloudflare DNS targets
Closes #issue-ipv6-only-environment
* fix(build): use SSH_AUTHORIZED_KEY from user defaults when no SSH_KEYS_FILE exists
When using 'User Defaults' with var_ssh_authorized_key set in default.vars,
the SSH key was not being installed because install_ssh_keys_into_ct() only
checked for SSH_KEYS_FILE (which is only created in Advanced Settings).
Now the function also checks SSH_AUTHORIZED_KEY and creates a temporary
SSH_KEYS_FILE if needed.
Fixes#11062
Added 'yarn run tsc' to both update and install scripts for Joplin Server to ensure TypeScript sources are compiled. Also removed an unused variable from build.func for code cleanup.
Added initialization for ONLINE_TEMPLATE and ONLINE_TEMPLATES arrays at the start of the create_lxc_container function to ensure variables are defined before use.
