Fixes#17456.
The main tricky part has to do with what values the attribute should
have. LDAP defines a Boolean as
Boolean = "TRUE" / "FALSE"
so ideally we'd always see exactly those values. However,
although the issue is now marked as resolved, the discussion in
https://pagure.io/freeipa/issue/1259 shows how this may not always be
respected - meaning it makes sense for us to be more liberal in
interpreting these values.
The support for bullseye was added in #17951
but it was not documented as bullseye was
frozen and did not have proper configuration
files, hence wasn't documented.
Since now bullseye is released as a stable
version, it's support can be documented.
We previously used `zulip-puppet-apply` with a custom config file,
with an updated PostgreSQL version but more limited set of
`puppet_classes`, to pre-create the basic settings for the new cluster
before running `pg_upgradecluster`.
Unfortunately, the supervisor config uses `purge => true` to remove
all SUPERVISOR configuration files that are not included in the puppet
configuration; this leads to it removing all other supervisor
processes during the upgrade, only to add them back and start them
during the second `zulip-puppet-apply`.
It also leads to `process-fts-updates` not being started after the
upgrade completes; this is the one supervisor config file which was
not removed and re-added, and thus the one that is not re-started due
to having been re-added. This was not detected in CI because CI added
a `start-server` command which was not in the upgrade documentation.
Set a custom facter fact that prevents the `purge` behaviour of the
supervisor configuration. We want to preserve that behaviour in
general, and using `zulip-puppet-apply` continues to be the best way
to pre-set-up the PostgreSQL configuration -- but we wish to avoid
that behaviour when we know we are applying a subset of the puppet
classes.
Since supervisor configs are no longer removed and re-added, this
requires an explicit start-server step in the instructions after the
upgrades complete. This brings the documentation into alignment with
what CI is testing.
Recommonmark is no longer maintained, and MyST-Parser is much more
complete.
https://myst-parser.readthedocs.io/
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit 30eaed0378 (#15001) incorrectly
inserted a different section between the anchor and the heading.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The auth attempt rate limit is quite low (on purpose), so this can be a
common scenario where a user asks their admin to reset the limit instead
of waiting. We should provide a tool for administrators to handle such
requests without fiddling around with code in manage.py shell.
These checks suffer from a couple notable problems:
- They are only enabled on staging hosts -- where they should never
be run. Since ef6d0ec5ca, these supervisor processes are only
run on one host, and never on the staging host.
- They run as the `nagios` user, which does not have appropriate
permissions, and thus the checks always fail. Specifically,
`nagios` does not have permissions to run `supervisorctl`, since
the socket is owned by the `zulip` user, and mode 0700; and the
`nagios` user does not have permission to access Zulip secrets to
run `./manage.py print_email_delivery_backlog`.
Rather than rewrite these checks to run on a cron as zulip, and check
those file contents as the nagios user, drop these checks -- they can
be rewritten at a later point, or replaced with Prometheus alerting,
and currently serve only to cause always-failing Nagios checks, which
normalizes alert failures.
Leave the files installed if they currently exist, rather than
cluttering puppet with `ensure => absent`; they do no harm if they are
left installed.
This commit replaces add_emoji_by_admins_only with
add_custom_emoji_policy in new_feature_tutorial.md
as the old boolean setting is replaced by a new
integer setting.
Recently, the need for individual Markdown templates for
every endpoint's OpenAPI page was removed, as they are now
auto-generated from OpenAPI data. Further, as a part of this
migration, several new fields and Markdown extensions were added.
This commit updates the documentation to reflect the changes that
have occured as a result of the migration.
With various edits by tabbott to clarify or simplify the documentation.
`request.client` is no longer valid since the ZulipRequestNotes change.
This update the documentation to reflect that. And it also makes it
recommend `check_send_webhook_message` in favor of
`check_send_stream_message`.
Strictly speaking, this sentence is talking about the IdP configuration,
while the backend is just GenericOpenIdConnectBackend, so the new
phrasing is more correct.
The script is added to upgrade steps for 20.04 and Buster because
those are the upgrades that cross glibc 2.28, which is most
problematic. It will also be called out in the upgrade notes, to
catch those that have already done that upgrade.
Previously, one needed to specifying all the HTTP status
codes that we want to render along with the operation,
but the primary use case just needs the responses of
all the status codes, and not just one.
This commit modifies the Markdown extension to render
all the responses of all status codes of a specified
operation in a loop.
Now that we are starting to link this pages from the landing page's
top navigation, it makes sense to have proper backlinks to the
homepage so that there is some continuity when the user clicks on
a link that takes them to a ReadTheDocs page from the main website.
I heard with a new contributor that they were really confused by this
page, and I believe the key problem was that we didn't define the
somewhat technical term "linter".
We extracted a bunch of code into the common notification_data.py
module, with a data class managing a lot of the logic; this updates
the documentation to cover that.
Fixes#17277.
The main limitation of this implementation is that the sync happens if
the user authing already exists. This means that a new user going
through the sign up flow will not have their custom fields synced upon
finishing it. The fields will get synced on their consecutive log in via
SAML in the future. This can be addressed in the future by moving the
syncing code further down the codepaths to login_or_register_remote_user
and plumbing the data through to the user creation process.
We detail that limitation in the documentation.
Using puppet modules from the puppet forge judiciously will allow us
to simplify the configuration somewhat; this specifically pulls in the
stdlib module, which we were already using parts of.
The old name `push_notify_user_ids` was misleading, because
it does not contain user ids which should be notified for
the current message, but rather user ids who have the online
push notifications setting enabled.
When the Tornado server is restarted during an upgrade, if
server has old events with the `push_notify_user_ids` fields,
the server will throw error after this rename. Hence, we need
to explicitly handle such cases while processing the event.
Updated the install documentation to include the explanation of the
two new install options `--postgresql-database-name` and
`--postgresql-database-user`.
This makes it parallel with deliver_scheduled_messages, and clarifies
that it is not used for simply sending outgoing emails (e.g. the
`email_senders` queue).
This also renames the supervisor job to match.
Make it easier to locate docs for how to build an integration by adding
a link to the docs at the top of the docs on how to document an
integration.
Tweaked by tabbott to adjust the whole first paragraph and make the
link more contextual.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore. Depending on these projects is
no longer viable for us.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
* Move the extended documentation of code blocks to a separate page.
* Merge "code playgrounds" documentation to be a section of that page.
* Document copy widget on code blocks.
* This commit changes how we refer to "```python" type syntax for code
blocks. Instead of being called a syntax highlighting label, this is
now referred to as a "language tag", since it serves both syntax
highlighting and playgrounds.
* Remap all the links.
* Advertise this new page in various places that previously did not have a link.
This new pages accomplishes several interrelated things:
* Documents that Zulip Cloud runs master and how that works.
* Documents policies on how long client apps are expected to support
old releases in our compatibility matrix.
* Removes the 3-years-stale roadmap article.
* Provides a central place to talk about different versions in Zulip.
* Provides a better place to link to from our "you need to upgrade" nag.
This content is not intended to be final, but should be finalized in
the next week or so.
Fixes#18322.
This page is documentation for our participation in GSoD, which was a
few years ago. We can resurrect it if we participate in the program
again, so there's no need to keep it as a live page.
Requesting external images is a privacy risk, so route all external
images through Camo.
Tweaked by tabbott for better test coverage, more comments, and to fix
bugs.
This allows access to be more configurable than just setting one
attribute. This can be configured by setting the setting
AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL.
This hits the unauthenticated Github API to get the list of tags,
which is rate-limited to 60 requests per hour. This means that the
tool can only be run 60 times per hour before it starts to exit with
errors, but that seems like a reasonable limit for the moment.
Update `docs/production/install.md` and
`docs/production/deployment.md` to document the install flags that can
be used as part of the installer more clearly.
Fixes#18122.
The show password feature is a functionality to
toggle the visibility of the password fields in forms
so that one can check if they have entered the correct
password or not. We implement this using an eye icon
toggling which converts input field type from password
to text and vice-versa.
Fixes part of #17301.
Using `supervisorctl stop all` to stop the server is not terribly
discoverable, and may stop services which are not part of Zulip
proper.
Add an explicit tool which only stops the relevant services. It also
more carefully controls the order in which services are stopped to
minimize lost requests, and maximally quiesce the server.
Locations which may be stopping _older_ versions of Zulip (without
this script) are left with using `supervisorctl stop all`.
Fixes#14959.
I have made `tools/setup/optimize-svg` do the SVG optimization
automatically rather than just telling you the command to run if they
need optimizing. This included adding a `--check` parameter to use in
CI to only check as we previously did rather than actually running the
optimization.
I have also made `tools/setup/optimize-svg` execute
`tools/setup/generate_integration_bots_avatars.py` once it has run the
optimization to ensure it is always ran.
This makes it one less command to run when creating an integration,
but also means that we catch instances where a PNG has just been
copied into the `static/images/integrations/bot_avatars` folder as the
only instance where this won't be run is if `optimize-svg` has not
been run which would be caught in CI.
Fixes#18183. Fixes#18184.
I have updated the remote development documentations to be more accurate
when it comes to developing on a Zulip Development Droplet to ensure
the user knows to access at `zulip.username.zulipdev.org`.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reduces the complexity of our dependency graph.
It also makes sub_store.get parallel to message_store.get.
For both you pass in the relevant id to get the
full validated object.
This change attempts to highlight WSL 2 as the default installation
method for Windows; that is currently much more reliable than Vagrant.
Further work is probably needed to complete this transition.
This documentation does not work and has not been used for years.
At this point, `provision` is sufficiently flexible in terms of
supporting different platforms that any future work will be to extend
it, rather than maintaining awkward manual installation documentation.
This commits adds instructions to bring up the
vagrant development server using the Hyper-V provider.
Additionally, this commits also removes the indication
that this guide for `non vagrant use` from the top of
the document. Also fixes a little grammatical error
under the `Newer versions of supported distributions`
heading.
Fixes#16994.
I have updated the path for the svgo module in the integrations
documentation as the other path would error out as `svgo` wasn't in
PATH, so have updated to use `yarn run svgo` instead.
This was used by the old native Zulip Android app
(zulip/zulip-android). That app has been undeveloped for enough years
that we believe it no longer functions; as a result, there's no reason
to keep a prototype API endpoint for it (that we believe never worked).
This makes it much more clear that this feature does JSON encoding,
which previously was only indicated in the documentation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
It does not seem like an official version supporting Webpack 4 (to say
nothing of 5) will be released any time soon, and we can reimplement
it in very little code.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We use GIPHY web SDK to create popover containing GIFs in a
grid format. Simply clicking on the GIFs will insert the GIF in the compose
box.
We add GIPHY logo to compose box action icons which opens the GIPHY
picker popover containing GIFs with "Powered by GIPHY"
attribution.
It looks like folks are accidentally generating requests for new
country-specific languages when they sign up. This change clarifies
the instructions so that we avoid these requests.
* repo => repository for more standard language.
* Delete 3 repeats of explaining the origin/upstream/local.
* Add some links.
* Update `git pull` language for rebase changes.
The only downside of this is that it makes it harder to control the
order of these tests; which isn't that important. And the structure
of naming each with its test order fundamentally requires renaming
files when adding/deleting tests, so if we want to control the default
test order, we'd be better off doing that by just hardcoding a list in
the test runner code.
This likely still needs work on updating the list of highlights, as
well as an editing pass, but we shouldn't need to read the whole
`git log --stat` again.
I have updated the `Documenting an Integration` documentation to be more
accurate on the usage of `tools/generate-integration-docs-screenshot` as
well as the requirement to fill out the `DOC_SCREENSHOT_CONFIG` section
of `zerver/lib/integrations.py` and a few other minor changes.
Development environment docs contains an outdated
statement stating that the non-vagrant setup can't
be used on windows, while it can now be used on windows by
using wsl. This commit fixes the statement and points
the link towards the non-vagrant setup process page.
Fixes#17721.
Follow up to #14768. This feature was already non-functional due to
.alert-display { display: none; }, and if we want to reimplement it,
we should do it using a modern library.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This mainly extracts a new module called
browser_history. It has much fewer dependencies
than hashchange.js, so any modules that just
need the smaller API from browser_history now
have fewer transitive dependencies.
Here are some details:
* Move is_overlay_hash to hash_util.
* Rename hashchange.update_browser_history to
brower_history.update
* Move go_to_location verbatim.
* Remove unused argument for exit_overlay.
* Introduce helper functions:
* old_hash()
* set_hash_before_overlay()
* save_old_hash()
We now have 100% line coverage on the extracted
code.
This adds an option for restricting a ldap user
to only be allowed to login into certain realms.
This is done by configuring an attribute mapping of "org_membership"
to an ldap attribute that will contain the list of subdomains the ldap
user is allowed to access. This is analogous to how it's done in SAML.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
Timestamps are logged automatically by GitHub Actions and can be
made visible using log settings easily. Hence we remove the
unnecessary timestamps here to make the logs look much cleaner.
We have disabled CircleCI and are using GitHub Actions for automated
testing.
docs: Changed context from CircleCI to Github Actions and wrote
some documentation specific to GH Actions.
tools: Replaced env checks for CIRCLECI with GITHUB_ACTION.
README: Use GitHub Actions build status badge.
GitHub Actions supports doing more than just CI,
and so in some contexts it's less obvious that we're
talking about just the CI if we refer to it instead of CircleCI.
In the documentation of the "Sending messages," path for the `check_message` and `do_send_message` function is not provided. So, I added the path of both for future contributors.
This is more broadly useful than for just Kandra; provide
documentation and means to install Smokescreen for stand-alone
servers, and motivate its use somewhat more.
This commit adds about Remote - SSH extension (in VS Code), which
helps us develop remotely by providing a similar interface as if we
are developing locally. We also simplify the documentation for RMate
to use the new standards.
Tweaked by tabbott for simplification/formatting.
Go to Recent Topics on "#", no hash and "#recent_topics".
Go to Recent Topics as the last destination for escape key.
Map `a` key to All messages and change its hash to
`#all_messages`.
These aren't relevant to most users, but in the interest of
transparency, we also don't want the existence of these to feel like a
secret. And maybe publishing their existence will result in folks who
we forget to add to these private streams asking about them.
This document is mainly an answer to a set of questions other
developers have been asking about Zulip's architecture and scalability
design. It's intended for developers working on Zulip, to help with
thinking about where to prioritize further efforts to optimize
scalability.
This commit adds the Zulip logo to the readthedocs
documentation by adding an .svg file (taken from
zulip.com) to the `docs/images` directory.
Morever, it removes the name `Zulip` which was
written at the top of the sidebar because the
logo already has that.
Also fix incorrect 2011 copyright years; the original Zulip, Inc. that
was merged into Dropbox, Inc. was incorporated in 2012, not 2011.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We now require keywords, so that there is no
pitfall for mixing up boolean parameters.
Positional parameters are basically evil
when you have a bunch of bools.
I also make user_profile the first argument.
Finally, the code is more diff-friendly.
Support for saving it in the session is dropped in django3, the cookie
is the mechanism that needs to be used. The relevant i18n code doesn't
have access to the response objects and thus needs to delegate setting
the cookie to LocaleMiddleware.
Fixes the LocaleMiddleware point of #16030.
As of Feb 15th 2019, Hipchat Cloud and Stride
have reached End Of Life and are no longer
supported by Atlassian. Since it is almost 2 years
now we can remove the migration guides.
It seems the Ubuntu base image we use now has a new enough VirtualBox
Guest Additions to trigger the ETXTBSY bug even when it’s not upgraded
by the vagrant-vbguest plugin. Provide and document a way to
downgrade it.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
We seem to periodically get contributors who rebase upstream commits
onto their pull request rather than the other way around, resulting in
a lot of GitHub noise. The PRs where this happens were made from
branches named master. We have always documented that you should work
on a feature branch, but not from this page; maybe this will help
reduce that kind of confusion.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We add navigating to user home inside WSL virtual disk as another
step as many users clone Zulip inside a mounted windows disk and
run into permission issues when running provision.
Steve asked me to remove this, since the tictactoe game was always
intended as a proof of concept. Now that we have poll and todo
widgets, the sample code for tictactoe has much less value.
We replace the content and type in test_widgets.py to maintain
coverage.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.
We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.
Fixes#16246.
These are respected by `urllib`, and thus also `requests`. We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
This provides a single reference point for all zulip.conf settings;
these mostly link out to the more complete documentation about each
setting, elsewhere.
Fixes#12490.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name. This is minor
cleanup, but enabled by the other renames in the previous commit.
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes. Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.
Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.
[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.
Tweaked by tabbott to improve the logging output and update documentation.
Fixes#16586.
Restarting servers is what can cause service interruptions, and
increase risk. Add all of the servers that we use to the list of
ignored packages, and uncomment the default allowed-origins in order
to enable unattended upgrades.
There was likely more dependency complexity prior to 97766102df, but
there is now no reason to require that consumers explicitly include
zulip::apt_repository.
Tim Abbott
Anders Kaseorg
Mateusz Mandera
Eeshan Garg
Philipp Seßner
Gaurav Pandey
Alex Vandiver
sahil839
Suyash Vardhan Mathur
PIG208
manavdesai27
Suyash Vardhan Mathur
Steve Howell
Abhijeet Prasad Bodas
Dinesh
Sean Yuan
akshatdalton
Gilbert Bishop-White
Adam Birds
Aman Agrawal
Matthew Wegner
Siddharth Asthana
Alya Abbott
Wesley Aptekar-Cassels
strifel
Jennifer Hwang
Riken Shah
IgnisDa
Janneke Morin
Sumanth V Rao
shanukun
Vishnu KS
yashika
Yashika Jotwani
smit_patel
Ganesh Pawar
Priyank Kumar
Strifel
Sourabh
Nikhil Maske
nikhilmaske-2001
aryanshridhar
Atharva Parkale
marchantia
Aditya Mitra
Harsh kumar
rht
marchantia
Anders Kaseorg
Ben Steinberg
YashRE42
ryanreh99
the-rdt
areebbeigh