This adds a new API for sending basic analytics data (number of users,
number of messages sent) from a Zulip server to the Zulip Cloud
central analytics database, which will make it possible for servers to
elect to have their usage numbers counted in published stats on the
size of the Zulip ecosystem.
A key part of this is the new helper, get_user_by_delivery_email. Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.
Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
This fixes a lot of spammy output of the form:
2018-11-27 17:46:48.279 INFO [zerver.lib.push_notifications] Sending push notification to user 46
when running populate_db, which is both confusing (since we're not
actually sending push notifications here) and spammy.
Now that we allow multiple users to have registered the same token, we
need to configure calls to unregister tokens to only query the
targeted user_id.
We conveniently were already passing the `user_id` into the push
notification bouncer for the remove API, so no migration for older
Zulip servers is required.
Previously, Zulip did not correctly handle the case of a mobile device
being registered with a push device token being registered for
multiple accounts on the same server (which is a common case on
zulipchat.com). This was because our database `unique` and
`unique_together` indexes incorrectly enforced the token being unique
on a given server, rather than unique for a given user_id.
We fix this gap, and at the same time remove unnecessary (and
incorrectly racey) logic deleting and recreating the tokens in the
appropriate tables.
There's still an open mobile app bug causing repeated re-registrations
in a loop, but this should fix the fact that the relevant mobile bug
causes the server to 500.
Follow-up work that may be of value includes:
* Removing `ios_app_id`, which may not have much purpose.
* Renaming `last_updated` to `data_created`, since that's what it is now.
But none of those are critical to solving the actual bug here.
Fixes#8841.
There are several situations in which we want to create a Customer and
stripe.Customer object before we really have a billing relationship with a
customer. The main one is giving non-profit or educational discounts.
random_api_key, the function we use to generate random tokens for API
keys, has been moved to zerver/lib/utils.py because it's used in more
parts of the codebase (apart from user creation), and having it in
zerver/lib/create_user.py was prone to cyclic dependencies.
The function has also been renamed to generate_api_key to have an
imperative name, that makes clearer what it does.
Now reading API keys from a user is done with the get_api_key wrapper
method, rather than directly fetching it from the user object.
Also, every place where an action should be done for each API key is now
using get_all_api_keys. This method returns for the moment a single-item
list, containing the specified user's API key.
This commit is the first step towards allowing users have multiple API
keys.
Stripe already returns an appropriate error in prod, and these checks are
just a hassle in tests.
Also fixes an error where the check for Plan.objects.exists() was missing
a "not".
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.
Fixes part of #10042.
As detailed in the documentation changes, this simplifies the
development workflow for doing UI work on the /stats pages.
The cost is a ~10% increase the time it takes to run `populate_db`,
which doesn't happen very often (and for most purposes manifests as a
1% increase in the time it takes to rebuild the database from scratch).
The main remaining todo for correctly populating
RealmAuditLog.requires_billing_update is supporting the de-seating (and
corresponding re-seating) that happens after being offline for two weeks.
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are
zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'
which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This reflects the changes to the default URL publicly
displayed to the user. It also changes the default
URL of the default test server outgoing webhook, which
prevented the test server flaskbotrc from working out
of the box.
We essentially stop running create_realm_internal_bots during
every provisioing and move its operations to run from populate db.
In fact to speed things up a bit we actually make populate db call the
funcs which create_realm_internal_bots calls behind the scenes.
Fixes: #9467.
Previously, the stream colors index i was accidentally a function only
of the user, so each user got the same color for all their streams.
This should provide a lot nicer-looking development environment
experience.
We've had this sort of logic for GCM for a long time; it's worth
adding for APNS as well.
Writing this is a bit of a reminder that I'm not a fan of how our unit
tests for push notifications work.
Makes announce stream `is_announcement_only` for the dev db for easier
manual testing. The default value for `is_announcement_only` in
`bulk_create_streams` is False.
Enforcing the unique constraint adds an unnecessary support burden for
figuring out who actually controls a given hostname, and in particular, for
verifying updates to the org id/key on a re-install of the Zulip server.
We flip the Stream "Rome" to be a web public stream. Also we add
attribute is_web_public in various stream dicts and in the
bulk_create_streams function of bulk_create.py responsible for
default stream creation in dev environment.
Apparently, this bot account was not properly being tagged as an API
super user in the test database; resulting in incorrect behavior if we
tried to send to a private stream in a test.
(Note that there seems to also be a similar issue in production, that
we don't understand the cause of; that is unrelated).
Issue #2088 asked for a wrapper to be created for
`create_stream_if_needed` (called `ensure_stream`) for the 25 times that
`create_stream_if_needed` is called and ignores whether the stream was
created. This commit replaces relevant occurences of
`create_stream_if_needed` with `ensure_stream`, including imports.
The changes weren't significant enough to add any tests or do any
additional manual testing.
The refactoring intended to make the API easier to use in most cases.
The majority of uses of `create_stream_if_needed` ignored the second
parameter.
Fixes: #2088.
To ensure that we have some basic data for custom profile settings,
in the `populate_db` data set, remove `options['test_suite']` check
for adding intial custom profile data.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.
Fixes: #6977.
This commit adds a generic function called check_send_webhook_message
that does the following:
* If a stream is specified in the webhook URL, it sends a stream
message, otherwise sends a PM to the owner of the bot.
* In the case of a stream message, if a custom topic is specified
in the webhook URL, it uses that topic as the subject of the
stream message.
Also, note that we need not test this anywhere except for the
helloworld webhook. Since helloworld is our default example for
webhooks, it is here to stay and it made sense that tests for a
generic function such as check_send_webhook_message be tested
with an actual generic webhook!
Fixes#8607.
This makes it easier to work on features that depend on messages
having been sent in the past (E.g. the date parts of recipient bars).
The new feature only works with --threads=1; since with the ~100
default messages that populate_db generates, the multi-threaded
feature shouldn't have significant performance impact (and it would be
tricky to make increasing timestamps work with the multi-threaded
model), it's reasonable to just set the default number of threads to 1
for now and have this timestamp-spreading feature only supported with
--threads=1.
Fixes#8277.
This completes the separation of our logic for managing Stripe
customers from the view code for the billing page.
As we add more features to our Customer model and to our Stripe
integration, we might further separate those two things; but for now
they're nearly synonymous and there's no problem in them being mixed
together.
Pull the code that talks to Stripe out into its own functions.
In a followup commit we'll move these to a separate file, as well
as the error-handling logic that remains in the view function
for now.
Also fix the translation markings: the translated string must be a
constant (e.g. a format string), or else translation is impossible.
Viewing with `-b` shows the few changes that happen in the logic
as it moves out of the view function; viewing without shows the
few changes in the rest of the view function.
Several changes:
* De-duplicate code for different error types.
* No need to list lots of error subtypes where we aren't treating
them differently; StripeError is the base class of them all.
* Unexpected, non-Stripe-related, exceptions we can handle in the normal
way. Just make them show up in the billing-specific log too.
* The Stripe client library already logs type, code, param, and message
before raising an error, so we don't need to repeat those; just add the
HTTP status code (because it's not there already and sure why not),
and the Python exception type the client library chose to raise
in case that makes things a bit easier to interpret.
Normal server admins will never run this code, and zulipchat.com will
have this information configured before users see it, so this message
is really just for development.
Stripe Checkout means using JS code provided by Stripe to handle
almost all of the UI, which is great for us.
There are more features we should add to this page and changes we
should make, but this gives us an MVP.
[greg: expanded commit message; fixed import ordering and some types.]
We'd rather this work be just executed immediately, rather than
queued, since queued events can confuse the queue workers if the
database is dropped and recreated repeatedly.
The 'simple' realm was super broken and confusing for new users. We
should replace this with having an easy way to make a new realm in
development, done properly.
Fixes#6116.
Since a user could use the same installation of the Zulip mobile app
with multiple Zulip servers, correct behavior is to allow reusing the
same token with multiple Zulip servers in the RemotePushDeviceToken
model.
While it might be useful to have created welcome-bot earlier in a
certain sense, it's definitely not a good idea in this populate_db
implementation, because doing so threw off the random initial
assignment of users to streams and thus broke the casper tests.
This makes the standard checkboxes 7% darker and makes the disabled
ones about 12% darker + 7% darker than they were before, to
increase visibility.
Fixes: #6331.
And it works!
A couple of things still to do:
* When a device token is no longer active, we'll get HTTP status 410.
We should then remove the token from the database so we don't keep
trying to push to it. This is fairly urgent.
* The library we're using has a nice asynchronous API, but this
version doesn't use it. This is OK now, but async will be
essential at scale.
Create a generator script to pull lines from a play, enhancing
random lines with emoji, Markdown and other flair.
With numerous contributions from Rein Zustand and Tim Abbott to finish
the project.
Fixes: #1666.
Tim Abbott
Anders Kaseorg
Vishnu Ks
Rishi Gupta
Yago González
Roman Godov
Max Nussenbaum
Shubham Dhama
Yashashvi Dave
Robert Hönig
Aditya Bansal
Sampriti Panda
Shubham Padia
Umair Khan
Preston Hansen
sinwar
Jack Weatherilt
Harshit Bansal
Eeshan Garg
Nick Al
Greg Price
neiljp (Neil Pilgrim)
rht
Brock Whittaker
Marco Burstein
Viraat Chandra
Greg Price
Darren Fix
Vishnu Ks