The new tools now have more concise, more parallel names:
- rebuild-dev-database
- rebuild-test-database
The actual implementations are still pretty different:
rebuild-dev-database:
mostly delegates to 5 management scripts
rebuild-test-database:
is a very thin wrapper for generate-fixtures
We'll try to clean that up a bit soon.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.
And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.
There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient. The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.
(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)
ASIDE:
Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":
node_modules
yarn.lock
zulip-emoji-cache
zulip-npm-cache
zulip-py3-venv
zulip-thumbor-venv
zulip-venv-cache
zulip-yarn
I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?
./var/webpack-cache
./var/mypy-cache
In `docs/subsystems/emoji.md` we say this:
```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```
I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.
And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
Also make sure our documentation for upgrading is reasonable for
Stretch => Buster.
Our reasoning for deprecating support for these releases is as follows:
* Ubuntu 16.04 Xenial reached desktop EOL last year; and will reach
EOL on the server in about a year.
* Debian Stretch will each EOL in 2020 (the precise date is unclear in
Debian's documentation, but based on past precedent it's in the next
few months, perhaps July 2020).
https://wiki.debian.org/DebianReleases#Production_Releases
* Both Ubuntu 16.04 and Debian Stretch use Python 3.5 as the system
Python, which will reach EOL in September 2020 (and we're already
seeing various third-party dependencies that we use drop support for
them).
* While there is LTS support for these older releases, it's not clear it's
going to be worth the added engineering effort for us to maintain EOL
releases of the base OSes that we support.
* We (now) have clear upgrade instructions for moving to Debian Buster
and Ubuntu 18.04.
This defends against cross-origin session fixation attacks. Renaming
the cookies means this one-time upgrade will have the unfortunate side
effect of logging everyone out, but they’ll get more secure sessions
in return.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Instructions were added by doing the setup on Ubuntu 18.04 WSL 2.
While the setup should be similar for other distributions supported by
our `./tools/provision` script inside WSL, it has not been tested.
Polished by tabbott.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This setting is being overridden by the frontend since the last
commit, and the security model is clearer and more robust if we don't
make it appear as though the markdown processor is handling this
issue.
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
I found the solution by simply trying out EMAIL_USE_SSL and it
worked. I had problems with sending emails (did not work at all, there
wasn't even a connection going on - I checked with tcpdump. Then I
found this: To use port 465, you need to call
smtplib.SMTP_SSL(). Currently, it looks like Django only uses
smtplib.SMTP() (source: https://code.djangoproject.com/ticket/9575).
Fixes#14350.
This returns us to a consistent logging format regardless of whether
the request is authenticated.
We also update some log examples in docs to be consistent with the new
style.
Those docs were outdated and no longer represantative of how things
work. upgrade-python-dependencies and unupgradable.json are no longer a
thing, so the entire paragraph about them should be removed.
Then the requirements/README.md file is refreshed a bit to be more
accurate.
For historical reasons we were creating Recipient
objects at some point in the typing-notifications
codepath. Now we just work with UserProfiles.
This removes some queries, as indicated by
the change to `len(queries)` in a couple of the
tests.
The one subtle thing that changes here is huddles.
If user 10 sends a typing notification that they
are talking to users 20 and 30, there might not
actually be a huddle for users 10/20/30, but
we were actually creating huddles on the fly!
There is no need to create huddles just for
typing notifications, since we don't even
share huddle ids with our clients. The clients
just infer the huddles.
Some of the code that gets killed off here as
somewhat "collateral damage" is some
defensive code related to formerly supporting streams
in typing indicators. The support for streams
was killed off almost as soon as we released
the feature, and the codepath is pretty clearly
user-centric at this point.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.
Unfortunately, we still have some older versions
of mobile that still send emails.
In this commit we fix non-user-facing things
like docs and tests to promote the user_ids
interface that has existed since about version
2.0 of the server.
One annoyance is that we documented the
typing endpoint with emails, instead of the
more modern user_ids, which may have delayed
mobile converting to user_ids (and which
certainly caused confusion). It's trivial
to update the docs, but we need to short
circuit one assertion in the openapi tests.
We also clean up the test structure for the
typing tests:
TypingHappyPathTest.test_start_to_another_user
TypingHappyPathTest.test_start_to_multiple_recipients
TypingHappyPathTest.test_start_to_self
TypingHappyPathTest.test_start_to_single_recipient
TypingHappyPathTest.test_stop_to_another_user
TypingHappyPathTest.test_stop_to_self
TypingValidateOperatorTest.test_invalid_parameter
TypingValidateOperatorTest.test_missing_parameter
TypingValidateUsersTest.test_argument_to_is_not_valid_json
TypingValidateUsersTest.test_bogus_user_id
TypingValidateUsersTest.test_empty_array
TypingValidateUsersTest.test_missing_recipient
TypingValidationHelpersTest.test_recipient_for_user_ids
TypingValidationHelpersTest.test_recipient_for_user_ids_non_existent_id
TypingLegacyMobileSupportTest.test_legacy_email_interface
This has for a while been our only active Google Groups mailing list,
and given that folks will guess security@ as our security contact, we
might as well just standardize on that.
Also tweak some ambiguous text; it wouldn't be appropriate for us to
issue a CVE for e.g. an operational issue only affecting us.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
- I fixed a typo with "lowerecase"
- I elaborated on the prefix before elaborating
on the rest of the message (i.e. went in correct
order).
- I split out the provision example (since we
talk about it some depth).
- I added more positive examples.
- I removed the distracting italics around the
good commit messages.
- I moved the "gather_subscriptions" commit to
the bottom of the list, since we elaborate
on that below the list.
This includes an experiment of having a draft of the 2.1.3 changelog,
which is helpful in avoiding duplication with the 2.2.0 changelog for
items we're planning to backport.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.
Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:
* urls: django.urls.resolvers.RegexURLPattern has been replaced by
django.urls.resolvers.URLPattern; affects OpenAPI code and related
features which re-parse Django's internals.
https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.
There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.
Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.
Fixes#10835.
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.
Update the backend to use emoji_codes.json too instead of the three
separate JSON files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This is a fairly involved set of changes, including changes that:
* Delete various legacy or semi-duplicated sections of testing.md.
Nobody needs to manually delete the postgres datbase anymore, as
reflected in the fact that the docs still mention postgres 9.1 from
Ubuntu Precise.
* Simplify the distracting heading section at the top of testing.md.
* Move content on manual testing to docs/development/using.md.
* Moves some content related to managing the database to
schema-migrations.md. (Resulting in some cleanups to that page as
well).
I ideally would have split this into smaller pieces.
Steve Howell
Puneeth Chaganti
Anders Kaseorg
Alex Vandiver
Tim Abbott
Vishnu Ks
arpit551
Anders Kaseorg
Rohitt Vashishtha
Steve Howell
Aman Agrawal
Ray Kraesig
thoemie
shubhamgupta2956
Mateusz Mandera
Ryan Rehman
Stefan Weil
Rike-Benjamin Schuppner
Yashwant
Vaibhav Raj Singh
rht
Chris Bobbe
Dinesh
xpac1985