- Updates instances of "private message", "PM", and "private_message",
excluding historical references in `overview/changelog.md`.
- Also excludes `/docs/translating` since we would need new
translations for "direct messages" and "DMs".
Previously, `X-Forwarded-Proto` did not need to be set, and failure to
set `loadbalancer.ips` would merely result in bad IP-address
rate-limiting and incorrect access logs; after 0935d388f0, however,
failure to do either of those, if Zulip is deployed with `http_only`,
will lead to infinite redirect loops after login. These are
accompanied by a misleading error, from Tornado, of:
Forbidden (Origin checking failed - https://zulip.example.com does not match any trusted origins.): /json/events
This is most common with Docker deployments, where deployments use
another docker container, such as nginx or Traefik, to do SSL
termination. See zulip/docker-zulip#403.
Update the documentation to reinforce that `loadbalancer.ips` also
controls trust of `X-Forwarded-Proto`, and that failure to set it will
cause the application to not function correctly.
- Create a dedicated "Reporting bugs" page to learly document
where and how bugs should be reported.
- Drop "Reporting issues" section from the Contributing guide.
- Delete "Bug report guidelines" page.
04cf68b45e make nginx responsible for downloading (and caching)
files from S3. As noted in that commit, nginx implements its own
non-blocking DNS resolver, since the base syscall is blocking, so
requires an explicit nameserver configuration. That commit used
127.0.0.53, which is provided by systemd-resolved, as the resolver.
However, that service may not always be enabled and running, and may
in fact not even be installed (e.g. on Docker). Switch to parsing
`/etc/resolv.conf` and using the first-provided nameserver. In many
deployments, this will still be `127.0.0.53`, but for others it will
provide a working DNS server which is external to the host.
In the event that a server is misconfigured and has no resolvers in
`/etc/resolv.conf`, it will error out:
```console
Error: Evaluation Error: Error while evaluating a Function Call, No nameservers found in /etc/resolv.conf! Configure one by setting application_server.nameserver in /etc/zulip/zulip.conf (file: /home/zulip/deployments/current/puppet/zulip/manifests/app_frontend_base.pp, line: 76, column: 70) on node example.zulipdev.org
```
This better presents the CSS organization for readers, and also
removes a stale reference and link to Bootstrap.
Because postcss-nesting's spec-aligned syntax has tripped up some
contributors, the mention of PostCSS now includes a link to the
postcss-nesting README and the CSS Nesting spec from the W3C, which
PostCSS Nesting attempts to adhere to.
This commit represents an in-place reordering of the document. No
headings or content has been changed (that will happen in subsequent
commits).
The goal is to open the document with generic advice and guidance
applicable to all Zulip developers across all languages:
1. Consistency, enforced by linters and automated tests, opens the
document.
2. General, largely language-neutral advice about line length,
third-party code, translation, paths, and secrets come next.
3. Next up is language-specific advice and conventions: Python,
followed by JavaScript and TypeScript, followed by HTML and CSS
(although the HTML and CSS will be moved in a subsequent commit
to their own file).
4. Closing the file, rather than opening it, is the section on
Dangerous constructs. Some of these are fairly specialized, so
it makes sense not to ask readers to read through them before
presenting, say, our philosophy on line length.
Finally, in trying to come up with a sensible order for all sections
of this document, the "More arbitrary style things" heading has been
removed.
This is a useful improvement in general for making correct
LogoutRequests to Idps and a necessary one to make SP-initiated logout
fully work properly in the desktop application. During desktop auth
flow, the user goes through the browser, where they log in through their
IdP. This gives them a logged in browser session at the IdP. However,
SAML SP-initiated logout is fully conducted within the desktop
application. This means that proper information needs to be given to the
the IdP in the LogoutRequest to let it associate the LogoutRequest with
that logged in session that was established in the browser. SessionIndex
is exactly the tool for that in the SAML spec.
This gives more flexibility on a server with multiple organizations and
SAML IdPs. Such a server can have some organizations handled by IdPs
with SLO set up, and some without it set up. In such a scenario, having
a generic True/False server-wide setting is insufficient and instead
being able to specify the IdPs/orgs for SLO is needed.
Closes#20084
This is the flow that this implements:
1. A logged-in user clicks "Logout".
2. If they didn't auth via SAML, just do normal logout. Otherwise:
3. Form a LogoutRequest and redirect the user to
https://idp.example.com/slo-endpoint?SAMLRequest=<LogoutRequest here>
4. The IdP validates the LogoutRequest, terminates its own user session
and redirects the user to
https://thezuliporg.example.com/complete/saml/?SAMLRequest=<LogoutResponse>
with the appropriate LogoutResponse. In case of failure, the
LogoutResponse is expected to express that.
5. Zulip validates the LogoutResponse and if the response is a success
response, it executes the regular Zulip logout and the full flow is
finished.
Django 4.0 and higher began checking the `Origin` header, which made
it important that Zulip know accurately if the request came over HTTPS
or HTTP; failure to do so would result in "CSRF verification failed"
errors.
For Zulip servers which are accessed via proxies, this means that
`X-Fowarded-Proto` must be set accurately. Adjust the documentation
for the suggested configurations to add the header.
Fixes: #24599.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
1c76036c61 raised the number of `minfds` in Supervisor from 40k to
1M. If Supervisor cannot guarantee that number of available file
descriptors, it will fail to start; `/etc/security/limits.conf` was
hence adjusted upwards as well. However, on some virtualized
environments, including Proxmox LXC, setting
`/etc/security/limits.conf` may not be enough to raise the
system-level limits. This causes `supervisord` with the larger
`minfds` to fail to start.
The limit of 1000000 was chosen to be arbitrarily high, assuming it
came without cost; it is not expected to ever be reached on any
deployment. 262b19346e already lowered one aspect of that
changeset, upon determining it did come with a cost. Potentially
breaking virtualized deployments during upgrade is another cost of
that change.
Lower the `minfds` it back down to 40k, partially reverting
1c76036c61, but allow adjusting it upwards for extremely large
deployments. We do not expect any except the largest deployments to
ever hit the 40k limit, and a frictionless deployment for the
vanishingly small number of huge deployments is not worth the
potential upgrade hiccups for the much more frequent smaller
deployments.
Instead of copying over a mostly-unchanged `postgresql.conf`, we
transition to deploying a `conf.d/zulip.conf` which contains the
only material changes we made to the file, which were previously
appended to the end.
While shipping separate while `postgresql.conf` files for each
supported version is useful if there is large variety in supported
options between versions, there is not no such variation at current,
and the burden of overriding the entire default configuration is that
it must be keep up to date wit the package's version.
This commit updates the recommended commit summary length
to 72 characters in the documentation. If the commit message
exceeds this length than GitHub cutoffs the remaining characters.
This should help miscueing users into thinking that the provisioning
steps, etc. are a part of the python3 installation--which is now more
explicitly aimed at Centos, Fedora, and RHEL users.
In #23380 we want to change all ocurrences of `uri` to `url`. This
commit changes the ocurrences of `uri` appeared in files related to
email, including templates (`.html`, `.txt`) and backend (`.py`)
codes.
In `email.md`, `base_images_uri` is changed to `images_base_url` -
the words `base` and `images` are swapped and plural form is added
for `image`. This is becasue the former is not found anywhere in
the codebase while the later appears a lot. To reduce confusion,
this doccumentation changed accordingly.
Taking backups on the database primary adds additional disk load,
which can impact the performance of the application.
Switch to taking backups on replicas, if they exist. Some deployments
may have multiple replicas, and taking backups on all of them is
wasteful and potentially confusing; add a flag to inhibit taking
nightly snapshots on the host.
If the deployment is a single instance of PostgreSQL, with no
replicas, it takes backups as before, modulo the extra flag to allow
skipping taking them.
Added unmute.svg in web/shared/icons. Also added
source and license information for the icon in
docs/THIRDPARTY.
Additionally, used unmute icon for unmute option in
topic_sidebar menu for topics in muted stream.
Fixes#25124
This commit make changes in create stream panel by moving
announce stream option below stream description and adds a
"Stream permissions" section heading just above
"Who can access the stream?" and also rewords the text
"Announce stream".
Also now announce stream option is only shown if the user creating
stream have access to the announcement stream name. When option
is not shown, default behaviour is to announce public and
web-public streams.
Fixes#23327.
This removes the production reporting to `/json/report/error` upon
`blueslip.error`, and replaces it with reporting to Sentry, if
enabled. Sentry provides better reporting and grouping for exceptions
than the email- and `#errors`-reporting provided by the
`/json/report/error` endpoint.
The development behaviour of rendering `blueslip.error` messages and
stacktraces immediately, and stopping execution, is preserved.
To better chain exception information, the whole previous exception is
passed to `blueslip.error`, not just the stack, and the second
parameter is formalized to be an object to map to Sentry's "context"
concept.
We currently use the GitHub logo to show the bots settings in both
personal and organization settings. This icon is not a correct
representation since we use have support for several different bots
including but not limited to a GitHub integration bot.
This commit replaces the GitHub logo with Smart Toy material icon
from Google, while also adding the required licencing details at
docs/THIRDPARTY.
Part of #19797.
Since logrotate runs in a daily cron, this practically means "daily,
but only if it's larger than 500M." For large installs with large
traffic, this is effectively daily for 10 days; for small installs, it
is an unknown amount of time.
Switch to daily logfiles, defaulting to 14 days to match nginx; this
can be overridden using a zulip.conf setting. This makes it easier to
ensure that access logs are only kept for a bounded period of time.
Previously, we had an architecture where CSS inlining for emails was
done at provision time in inline_email_css.py. This was necessary
because the library we were using for this, Premailer, was extremely
slow, and doing the inlining for every outgoing email would have been
prohibitively expensive.
Now that we've migrated to a more modern library that inlines the
small amount of CSS we have into emails nearly instantly, we are able
to remove the complex architecture built to work around Premailer
being slow and just do the CSS inlining as the final step in sending
each individual email.
This has several significant benefits:
* Removes a fiddly provisioning step that made the edit/refresh cycle
for modifying email templates confusing; there's no longer a CSS
inlining step that, if you forget to do it, results in your testing a
stale variant of the email templates.
* Fixes internationalization problems related to translators working
with pre-CSS-inlined emails, and then Django trying to apply the
translators to the post-CSS-inlined version.
* Makes the send_custom_email pipeline simpler and easier to improve.
Signed-off-by: Daniil Fadeev <fadeevd@zulip.com>
Primary goal of library replacement is improving execution speed.
This commit should not affect the functionality of the system
or make any changes to it.
This adds a new search icon which we prefer over the one made
available from bootstrap, and replaces search icons in navbar
search with the Ionic icon.
Increasing worker_connections has a memory cost, unlike the rest of
the changes in 1c76036c61d8; setting it to 1 million caused nginx to
consume several GB of memory.
Reduce the default down to 10k, and allow deploys to configure it up
if necessary. `worker_rlimit_nofile` is left at 1M, since it has no
impact on memory consumption.
Updates this section to use the shared `api-doc-template.md` as a
guide, as well as the current documentation for the `render-message`
endpoint. Previously, this section referred to a file that had been
removed with the transition to a shared template file.
Fixes#24485.
Adds a point in the "Tips and best practices" section of the
article on writing reviewable pull requests about cross-linking
between GitHub pull requests and conversations in the Zulip
development community.
Fixes a few typos / errors in commit a633890d8 that updated
these documentation articles on pull requests.
Also adjusts language around screenshots to match what we use
in the pull request template on GitHub for the zulip server and
web app repository.
Zulip Server 4.0 is now 22 months old, which is more than 18 months.
Per the general policy in the "Client apps" section below, that
means it's time to drop support for older versions.
We released 5.0 near the end of 2022-03, so near the end of 2023-09
we can update this further to say 5.0.
Since we are migrating from JavaScript to TypeScript
some files in the documentation are still with their
old extension. This commit changes those file extensions.
Adds a 'Tips and best practices' section to the documentation on
writing reviewable pull requests.
Also, updates step 3 of the documentation on creating pull requests
to link to the new section and to not have an out-of-date screenshot
of the GitHub pull request template.
Because `yarn.lock` includes transitive dependencies, it already pins
our dependencies more comprehensively than `package.json` would if we
followed this bad advice, which we don’t, as of commit
9b0401b76d (#13118).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is the behaviour inherited from Django[^1]. While setting the
password to empty (`email_password = `) in
`/etc/zulip/zulip-secrets.conf` also would suffice, it's unclear what
the user would have been putting into `EMAIL_HOST_USER` in that
context.
Because we previously did not warn when `email_password` was not
present in `zulip-secrets.conf`, having the error message clarify the
correct configuration for disabling SMTP auth is important.
Fixes: #23938.
[^1]: https://docs.djangoproject.com/en/4.1/ref/settings/#std-setting-EMAIL_HOST_USER
The documentation for restoring backups referenced that it needed to
be to the same version of PostgreSQL, but did not explain how to do
that.
Link to the relevant section of the installer documentation, and name
the flag explicitly.
Fixes: #23691
This lets us simplify the long-ish ‘../../static/js’ paths, and will
remove the need for the ‘zrequire’ wrapper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Ever since we started bundling the app with webpack, there’s been less
and less overlap between our ‘static’ directory (files belonging to
the frontend app) and Django’s interpretation of the ‘static’
directory (files served directly to the web).
Split the app out to its own ‘web’ directory outside of ‘static’, and
remove all the custom collectstatic --ignore rules. This makes it
much clearer what’s actually being served to the web, and what’s being
bundled by webpack. It also shrinks the release tarball by 3%.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These hooks are run immediately around the critical section of the
upgrade. If the upgrade fails for preparatory reasons, the pre-deploy
hook may not be run; if it fails during the upgrade, the post-deploy
hook will not be run. Hooks are called from the CWD of the new
deploy, with arguments of the old version and the new version. If
they exit with non-0 exit code, the deploy aborts.
Corepack manages multiple per-project version of Yarn and PNPM, which
means we have to maintain less installation code, and could help us
switch away from Yarn 1 without making the system unusable for
development of other Yarn 1 projects.
https://nodejs.org/api/corepack.html
The Unicode spaces in the timerender test resulted from an ICU
upgrade: https://github.com/nodejs/node/pull/45068.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We will hopefully be able to just this in #16208 to document what
users need to configure in order to do this manually, but the content
here will be useful for anyone who hasn't set that up regardless.
This overrides the settings in the web interface, and enables much
more flexible configuration of the build environment (which previously
defaulted to Python 3.7).
https://docs.readthedocs.io/en/stable/config-file/v2.html
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This adds a new endpoint /jwt/fetch_api_key that accepts a JWT and can
be used to fetch API keys for a certain user. The target realm is
inferred from the request and the user email is part of the JWT.
A JSON containing an user API key, delivery email and (optionally)
raw user profile data is returned in response.
The profile data in the response is optional and can be retrieved by
setting the POST param "include_profile" to "true" (default=false).
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
- Updates `.prettierignore` for the new directory.
- Updates any reference to the API documentation directory for
markdown files to be `api_docs/` instead of `zerver/api/`.
- Removes a reference link from `docs/documentation/api.md` that
hasn't referenced anything in the text since commit 0542c60.
- Update rendering of API documentation for new directory.
- Renames "Customize Zulip" to "Server configuration".
- Cross-links "Server configuration" with "System and deployment
configuration".
Fixes part of #23984.
Karl Stolley
Matthew Volpe
David Rosa
Alex Vandiver
Anders Kaseorg
Alya Abbott
Tim Abbott
Zixuan James Li
Lauryn Menard
Mateusz Mandera
Toyam Cox
sbansal1999
Karl Stolley
AcKindle3
Hardik Dharmani
Ujjawal Modi
sayamsamal
Daniil Fadeev
YashRE42
oribe1115
Alya Abbott
Chris Bobbe
Lakshay Mittal
Alessandro Toppi
Dhruv Gupta