We essentially stop running create_realm_internal_bots during
every provisioing and move its operations to run from populate db.
In fact to speed things up a bit we actually make populate db call the
funcs which create_realm_internal_bots calls behind the scenes.
Fixes: #9467.
Previously, the stream colors index i was accidentally a function only
of the user, so each user got the same color for all their streams.
This should provide a lot nicer-looking development environment
experience.
We've had this sort of logic for GCM for a long time; it's worth
adding for APNS as well.
Writing this is a bit of a reminder that I'm not a fan of how our unit
tests for push notifications work.
Makes announce stream `is_announcement_only` for the dev db for easier
manual testing. The default value for `is_announcement_only` in
`bulk_create_streams` is False.
Enforcing the unique constraint adds an unnecessary support burden for
figuring out who actually controls a given hostname, and in particular, for
verifying updates to the org id/key on a re-install of the Zulip server.
We flip the Stream "Rome" to be a web public stream. Also we add
attribute is_web_public in various stream dicts and in the
bulk_create_streams function of bulk_create.py responsible for
default stream creation in dev environment.
Apparently, this bot account was not properly being tagged as an API
super user in the test database; resulting in incorrect behavior if we
tried to send to a private stream in a test.
(Note that there seems to also be a similar issue in production, that
we don't understand the cause of; that is unrelated).
Issue #2088 asked for a wrapper to be created for
`create_stream_if_needed` (called `ensure_stream`) for the 25 times that
`create_stream_if_needed` is called and ignores whether the stream was
created. This commit replaces relevant occurences of
`create_stream_if_needed` with `ensure_stream`, including imports.
The changes weren't significant enough to add any tests or do any
additional manual testing.
The refactoring intended to make the API easier to use in most cases.
The majority of uses of `create_stream_if_needed` ignored the second
parameter.
Fixes: #2088.
To ensure that we have some basic data for custom profile settings,
in the `populate_db` data set, remove `options['test_suite']` check
for adding intial custom profile data.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.
Fixes: #6977.
This commit adds a generic function called check_send_webhook_message
that does the following:
* If a stream is specified in the webhook URL, it sends a stream
message, otherwise sends a PM to the owner of the bot.
* In the case of a stream message, if a custom topic is specified
in the webhook URL, it uses that topic as the subject of the
stream message.
Also, note that we need not test this anywhere except for the
helloworld webhook. Since helloworld is our default example for
webhooks, it is here to stay and it made sense that tests for a
generic function such as check_send_webhook_message be tested
with an actual generic webhook!
Fixes#8607.
This makes it easier to work on features that depend on messages
having been sent in the past (E.g. the date parts of recipient bars).
The new feature only works with --threads=1; since with the ~100
default messages that populate_db generates, the multi-threaded
feature shouldn't have significant performance impact (and it would be
tricky to make increasing timestamps work with the multi-threaded
model), it's reasonable to just set the default number of threads to 1
for now and have this timestamp-spreading feature only supported with
--threads=1.
Fixes#8277.
This completes the separation of our logic for managing Stripe
customers from the view code for the billing page.
As we add more features to our Customer model and to our Stripe
integration, we might further separate those two things; but for now
they're nearly synonymous and there's no problem in them being mixed
together.
Pull the code that talks to Stripe out into its own functions.
In a followup commit we'll move these to a separate file, as well
as the error-handling logic that remains in the view function
for now.
Also fix the translation markings: the translated string must be a
constant (e.g. a format string), or else translation is impossible.
Viewing with `-b` shows the few changes that happen in the logic
as it moves out of the view function; viewing without shows the
few changes in the rest of the view function.
Several changes:
* De-duplicate code for different error types.
* No need to list lots of error subtypes where we aren't treating
them differently; StripeError is the base class of them all.
* Unexpected, non-Stripe-related, exceptions we can handle in the normal
way. Just make them show up in the billing-specific log too.
* The Stripe client library already logs type, code, param, and message
before raising an error, so we don't need to repeat those; just add the
HTTP status code (because it's not there already and sure why not),
and the Python exception type the client library chose to raise
in case that makes things a bit easier to interpret.
Normal server admins will never run this code, and zulipchat.com will
have this information configured before users see it, so this message
is really just for development.
Stripe Checkout means using JS code provided by Stripe to handle
almost all of the UI, which is great for us.
There are more features we should add to this page and changes we
should make, but this gives us an MVP.
[greg: expanded commit message; fixed import ordering and some types.]
We'd rather this work be just executed immediately, rather than
queued, since queued events can confuse the queue workers if the
database is dropped and recreated repeatedly.
The 'simple' realm was super broken and confusing for new users. We
should replace this with having an easy way to make a new realm in
development, done properly.
Fixes#6116.
Since a user could use the same installation of the Zulip mobile app
with multiple Zulip servers, correct behavior is to allow reusing the
same token with multiple Zulip servers in the RemotePushDeviceToken
model.
While it might be useful to have created welcome-bot earlier in a
certain sense, it's definitely not a good idea in this populate_db
implementation, because doing so threw off the random initial
assignment of users to streams and thus broke the casper tests.
This makes the standard checkboxes 7% darker and makes the disabled
ones about 12% darker + 7% darker than they were before, to
increase visibility.
Fixes: #6331.
And it works!
A couple of things still to do:
* When a device token is no longer active, we'll get HTTP status 410.
We should then remove the token from the database so we don't keep
trying to push to it. This is fairly urgent.
* The library we're using has a nice asynchronous API, but this
version doesn't use it. This is OK now, but async will be
essential at scale.
Create a generator script to pull lines from a play, enhancing
random lines with emoji, Markdown and other flair.
With numerous contributions from Rein Zustand and Tim Abbott to finish
the project.
Fixes: #1666.
This system hasn't been in active use for several years, and had some
problems with it's design. So it makes sense to just remove it to declutter
the codebase.
Fixes#5655.
This prevents a buggy old Zulip server from forwarding invalid-format
push notification tokens to the push notification bouncer service.
As part of this change, we switch the token from Text to str to match
the rest of the code path.
Once we implement org_type-specific features, it'll be easy to change a
corporate realm to a community realm, but hard to go the other way. The main
difference (the main thing that makes migrating from a community realm to a
corporate realm hard) is that you'd have to make everyone sign another terms
of service.
This is an incomplete cleaned-up continuation of Lisa Neigut's push
notification bouncer work. It supports registration and
deregistration of individual push tokens with a central push
notification bouncer server.
It still is missing a few things before we can complete this effort:
* A registration form for server admins to configure their server for
this service, with tests.
* Code (and tests) for actually bouncing the notifications.
This feature has been obsolete since when Zulip was released as open
source software, since it's purpose was to avoid putting a "server
url" prompt in the desktop app, and now that prompt is required
anyway.
Change `from django.utils.timezone import now` to
`from django.utils import timezone`.
This is both because now() is ambiguous (could be datetime.datetime.now),
and more importantly to make it easier to write a lint rule against
datetime.datetime.now().
This fixes an issue where one would get errors of the form:
`ValueError: unsupported pickle protocol: 3`
in a `run-dev.py` server run against Python 2 if you ran `provision`.
Provision currently runs `populate_db` with Python 3, storing Python 3
based data in memcached, which then can't be read by Python 2.
The realm with string_id of "simple" just has three users
named alice, bob, and cindy for now. It is useful for testing
scenarios where realms don't have special zulip.com exception
handling.
This old helper has for years been used only by populate_db, and got
buggy (as of a recent refactoring). So we just call do_send_messages
directly instead.
Fixes the provisioning error we currently get in Travis CI.
A lot of care has been taken to ensure we're using the realm that the
message is being sent into, not the realm of the sender, to correctly
handle the logic for cross-realm bot users such as the notifications
bot.
Finishes the refactoring started in c1bbd8d. The goal of the refactoring is
to change the argument to get_realm from a Realm.domain to a
Realm.string_id. The steps were
* Add a new function, get_realm_by_string_id.
* Change all calls to get_realm to use get_realm_by_string_id instead.
* Remove get_realm.
* (This commit) Rename get_realm_by_string_id to get_realm.
Part of a larger migration to remove the Realm.domain field entirely.
First step in cleaning up populate_db.create_streams and
bulk_create.bulk_create_streams. Part of a series of commits to remove
Realm.domain from populate_db.
We are prone to case-sensitivity bugs, so I added AARON and ZOE.
Also, for good measure, I insert them in non-alphabetical order
to try to drive out bugs from non-consistent sorting of user ids.
This adds a couple new tools that can be used to determine whether a
particular change in Zulip's backend markdown processor would impact
the rendering of historical messages, without a human actually looking
at the message content. This is a useful way to verify whether a
change to our markdown syntax is likely to create problems.
[commit message and code tweaked by tabbott]
Previously, we set restrict_to_domain and invite_required differently
depending on whether we were setting up a community or a corporate
realm. Setting restrict_to_domain requires validation on the domain of the
user's email, which is messy in the web realm creation flow, since we
validate the user's email before knowing whether the user intends to set up
a corporate or community realm. The simplest solution is to have the realm
creation flow impose as few restrictions as possible (community defaults),
and then worry about restrict_to_domain etc. after the user is already in.
We set the test suite to explictly use the old defaults, since several of
the tests depend on the old defaults.
This commit adds a database migration.
Does a database migration to rename Realm.subdomain to
Realm.string_id, and makes Realm.subdomain a property. Eventually,
Realm.string_id will replace Realm.domain as the handle by which we
retrieve Realm objects.
This is a preliminary step towards eliminating the realm.domain field
in favor of realm.subdomain. Includes a database migration to create
these for existing realms.
The command to render old messages now looks for all messages
not matching the bugdown version, and it no longer directly calls
into model code. We should still be extremely cautious about
using this code.
This adds support for running a Zulip production server with each
realm on its own unique subdomain, e.g. https://realm_name.example.com.
This patch includes a ton of important features:
* Configuring the Zulip sesion middleware to issue cookier correctly
for the subdomains case.
* Throwing an error if the user tries to visit an invalid subdomain.
* Runs a portion of the Casper tests with REALMS_HAVE_SUBDOMAINS
enabled to test the subdomain signup process.
* Updating our integrations documentation to refer to the current subdomain.
* Enforces that users can only login to the subdomain of their realm
(but does not restrict the API; that will be tightened in a future commit).
Note that toggling settings.REALMS_HAVE_SUBDOMAINS on a live server is
not supported without manual intervention (the main problem will be
adding "subdomain" values for all the existing realms).
[substantially modified by tabbott as part of merging]
When we refactored zilencer to use a single urls.py file in
bf50dd7771, we accidentally lost the
prefix on the API urls.
This broke sending error report emails if zilencer was enabled.
For a long time, rest_dispatch has had this hack where we have to
create a copy of it in each views file using it, in order to directly
access the globals list in that file. This removes that hack, instead
making rest_dispatch just use Django's import_string to access the
target method to use.
[tweaked and reorganized from acrefoot's original branch in various
ways by tabbott]
Also fix the annotation of zilencer.views.report_error.
The `report` arguments are a Dict containing both strings and the
`more_info` sub-dictionary, so we type them as Dict[str, Any].
[tweaked by tabbott]
[Substantially revised by tabbott]
This probably still has some bugs in it, but having mostly complete
annotations for models.py will help a lot for the annotations folks
are adding to other files.
The previous separated-out configuration wasn't helping us, and this
makes it easier to make the extra installed applications pluggable in
the following commits.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.
As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.
When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.
We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.
Fixes#441.
Previously, the UserProfile objects were created in the order
generated by a Set, which meant tests would randomly start failing if
the code that runs before this part of populate_db changed (and thus
caused the Set object used to pass users into bulk_create_users to
have a different order when enumerated).
This fixes the issue in two ways -- one by sorting the users inside
bulk_create_users, and second by attaching subscriptions to users
based on a deterministic ordering.
get_realm is better in two key ways:
* It uses memcached to fetch the data from the cache and thus is faster.
* It does a case-insensitive query and thus is more safe.
This commit loses some indexes, unique constraints etc. that were
manually added by the old migrations. I plan to add them to a new
migration in a subsequent commit.
(imported from commit 4bcbf06080a7ad94788ac368385eac34b54623ce)
Vishnu Ks
Aditya Bansal
Tim Abbott
Sampriti Panda
Shubham Padia
Rishi Gupta
Yashashvi Dave
Umair Khan
Shubham Dhama
Preston Hansen
sinwar
Jack Weatherilt
Harshit Bansal
Eeshan Garg
Nick Al
Greg Price
neiljp (Neil Pilgrim)
rht
Brock Whittaker
Marco Burstein
Viraat Chandra
derAnfaenger
Greg Price
Darren Fix
Vishnu Ks
Vaida
vaibhav
Christian Hudon
Raghav Jajodia
Steve Howell
Tommy Ip
Rafid Aslam
reyha
Juan Verhook
Joy Chen
anirudhjain75
nikolay
bulat22101
Alex Huang
AZtheAsian
Anders Kaseorg
paxapy
Tomasz Kolek
Rishi Gupta
Sahil Dua
Taranjeet Singh
Tim Abbott
Eklavya Sharma
acrefoot
Evan Palmer
pxhanus
medullaskyline
Ashish Kumar
Varshit
Steven Oud
Tim Abbott
Reid Barton
David Roe