Commit Graph

822 Commits

Author SHA1 Message Date
Aman Agrawal a7eef1ed35 mark_all_messages_unread: Remove reset of depricated pointer. 2020-07-06 15:47:27 -07:00
Anders Kaseorg 6c9c12ee2d add_new_realm: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Tim Abbott 866965f162 views: Rename messages.py to message_fetch.py.
This completes to effort to split zerver.views.messages into smaller
modules focused on the different core type of work.
2020-06-22 15:08:34 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg 1f565a9f41 timezone: Use standard library datetime.timezone.utc consistently.
datetime.timezone is available in Python ≥ 3.2.  This also lets us
remove a pytz dependency from the PostgreSQL scripts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:34:17 -07:00
Sahil Batra 77d4be56a4
users: Modify do_create_user and create_user to accept role.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
2020-06-02 16:11:36 -07:00
sahil839 2fe8971204 populate_db: Change role of 'desdemona' to realm owner.
Modified by tabbott to update PROVISION_VERSION.
2020-06-01 15:22:27 -07:00
anshgit101 d0cfaebb15 populate_db: Reduce 'boring users' threshold. 2020-05-30 10:26:56 -04:00
sahil839 1aebf3cab9 actions: Merge do_change_is_admin and do_change_is_guest.
This commit merges do_change_is_admin and do_change_is_guest to a
single function do_change_user_role which will be used for changing
role of users.

do_change_is_api_super_user is added as a separate function for
changing is_api_super_user field of UserProfile.
2020-05-25 16:17:10 -07:00
sahil839 9b78a73e36 populate_db: Add new admin user as 'Desdemona'.
This commit adds a second admin user named 'Desdemona' to dev and
test database.
2020-05-19 11:42:27 -07:00
wowol 049288eaeb urls: Migrate zilencer urls to use modern django pattern. 2020-05-07 16:28:58 -07:00
Aman Agrawal a4495dbc6b populate_db: Generate topics using config fixture.
Instread of using stream_name + Intergers as topics, we now
generate topics using pos in `config.generate_data.json`.

This helps us create and test more realistic topics.
2020-05-05 16:11:09 -07:00
Aman Agrawal 8e0f0d2096 populate_db: Evenly distribute messages across all streams.
Populate db with messages after non-Shakespeare streams
are created so that we can distribute messages across them too.
2020-05-04 09:41:31 -07:00
Aman Agrawal b25f2ed5b3 populate_db: Add option to specify max no of topics to create.
This will help us create more topics per stream than we currently do
by default.
2020-05-04 09:41:31 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Steve Howell eb9a252ec9 populate_db, tests: Restrict emails in zulip realm.
We now restrict emails on the zulip realm, and now
`email` and `delivery_email` will be different for
users.

This change should make it more likely to catch
errors where we leak delivery emails or use the
wrong field for lookups.
2020-03-19 16:21:31 -07:00
Mateusz Mandera 89394fc1eb middleware: Use request.user for logging when possible.
Instead of trying to set the _requestor_for_logs attribute in all the
relevant places, we try to use request.user when possible (that will be
when it's a UserProfile or RemoteZulipServer as of now). In other
places, we set _requestor_for_logs to avoid manually editing the
request.user attribute, as it should mostly be left for Django to manage
it.
In places where we remove the "request._requestor_for_logs = ..." line,
it is clearly implied by the previous code (or the current surrounding
code) that request.user is of the correct type.
2020-03-09 13:54:58 -07:00
Mateusz Mandera fe0f381914 populate_db: Don't restrict email domains by default in tests and dev.
The email domain restriction to @zulip.com is annoying in development
environment when trying to test sign up. For consistency, it's best to
have tests use the same default, and the tests that require domain
restriction can be adjusted to set that configuration up for themselves
explicitly.
2020-03-07 18:38:59 -08:00
Steve Howell c4e3cfebb0 presence: Add realm_id to UserPresence.
This index is intended to optimize the performance of the very
frequently run query of "what is the presence status of all users in a
realm?".

Main changes:
    - add realm_id to UserPresence
    - add index for realm_id
    - backfill realm_id for old rows
    - change all writes to UserPresence to include
      realm_id

The index is of this form:

    "zerver_userpresence_realm_id_5c4ef5a9" btree (realm_id)

We will create an index on (realm_id, timestamp) in a
future commit, but I think it's a bit faster if you do
the backfill before the index.

There's also a minor tweak to the populate_db script.
2020-02-10 17:21:45 -08:00
Steve Howell 1f62b5e671 populate_db: Remove create_user_presenses() helper.
The last caller to this function was removed in
64230b3bd5

So it's been dead code for a while.
2020-02-10 17:20:44 -08:00
Ryan Rehman 174b2abcfd settings: Migrate to stream_post_policy structure.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.

It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.

This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.

With many tweaks by tabbott to documentation under /help, etc.

Fixes #13616.
2020-02-04 17:08:08 -08:00
Mateusz Mandera d24936cbe3 server_initialization: Use tos_version argument in create_users. 2020-01-29 17:26:45 -08:00
Mateusz Mandera 261da5999d populate_db: Extract default client creation to server_initialization. 2020-01-29 17:26:45 -08:00
Mateusz Mandera a25f00a69c populate_db: Extract some functions to server_initialization.py. 2020-01-29 17:26:45 -08:00
Vishnu KS 05b4610381 bots: Remove feedback cross realm bot.
This completes the remaining pieces of removing this missed in
d70e799466 (mostly in tests).
2020-01-25 22:54:44 -08:00
Anders Kaseorg cdda983e90 settings: Support optional memcached authentication.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Tim Abbott 8e7ce7cc79 python: Sort migrations/management command imports with isort.
This is a preparatory commit for using isort for sorting all of our
imports, merging changes to files where we can easily review the
changes as something we're happy with.

These are also files with relatively little active development, which
means we don't expect much merge conflict risk from these changes.
2020-01-14 13:07:47 -08:00
Tim Abbott 0dc7542363 populate_db: Fix cache flushing when rebuilding test database.
This fixes a similar problem to the last commit; we don't use
memcached with the test database, so we don't need to flush memcached
when rebuilding it.

(And if we try, we'll get exceptions trying to access the relevant
settings).
2020-01-13 18:22:55 -08:00
Tim Abbott 571ce2f5cb populate_db: Fix handling of memcached flushing.
Our recent fixes to using the system's configured memcached settings
broke populate_db, because its hacky clear_database helper is called
with a hacked-up settings module.

We fix this by first moving this out-of-place code from models.py into
populate_db, and then saving the settings required to access memcached
so that we can use them in clear_database.

We also fix a mypy erorr in flush-memcached that matches the same
issue fixed in clear_database.
2020-01-13 18:05:21 -08:00
Steve Howell 6a9eaebff2 populate_db: Make num_recips calculation more clear.
Extracting this calculation makes it easier to hack
it when you're trying to load lots of users.

We probably want a slightly more realistic calculation
here for stress testing.  And also fewer rows.  But
at least now it's a little more clear what it's doing.
2019-12-28 10:56:03 -08:00
Steve Howell 4a8c70593f populate_db: Add random names for large user batches.
If extra_users > 1000, add some names that are more
interesing than Extra111 User.
2019-12-28 10:56:03 -08:00
Tim Abbott 6407d0b1f9 push_notifications: Clear PushDeviceToken on API key change.
This includes adding a new endpoint to the push notification bouncer
interface, and code to call it appropriately after resetting a user's
personal API key.

When we add support for a user having multiple API keys, we may need
to add an additional key here to support removing keys associated with
just one client.
2019-11-19 15:37:43 -08:00
Anders Kaseorg cafac83676 request: Tighten type checking on REQ.
Then, find and fix a predictable number of previous misuses.

With a small change by tabbott to preserve backwards compatibility for
sending `yes` for the `forged` field.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:35:55 -08:00
Anders Kaseorg b0a7b33f9b push_notifications: Declare token of type str, not bytes.
Declaring a CharField of type bytes made no sense.

Signed-off-by: Anders Kaseorg <andersk@zulipchat.com>
2019-11-12 23:21:20 -08:00
Rishi Gupta 360cd7f147 remote data: Send RealmAuditLog data. 2019-10-08 17:27:29 -07:00
Rishi Gupta 48dc1d1128 remote data: Refactor remote_server_post_analytics to be more generic.
One small change in behavior is that this creates an array with all the
row_objects at once, rather than creating them 1000 at a time.

That should be fine, given that the client batches these in units of
10000 anyway, and so we're just creating 10K rows of a relatively
small data structure in Python code here.
2019-10-06 16:55:41 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Mateusz Mandera 4166c901ef do_update_user_custom_profile_data: Rename to ..._if_changed.
This adds clarity to the fact that the function no longer does
anything if the field values haven't changed.
2019-10-01 13:52:43 -07:00
Yashashvi Dave e05429b6a9 custom fields: Add default external account custom fields. 2019-08-28 15:35:53 -07:00
Mateusz Mandera 16479fa606 populate_db: Extract internal realm creation into a function. 2019-08-23 12:57:24 -07:00
Gloria Elston 092662d83e api: Use new endpoint for handling reactions. 2019-08-16 14:36:15 -07:00
Anders Kaseorg 5d063910ff zilencer: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Mateusz Mandera 8e0b6fa6cf populate_db: Use separate realm for system bots, like in production.
This commit alone breaks things, needs to be merged with the follow-up
ones.

welcome-bot is removed from the explicit list, because it already is in
settings.INTERNAL_BOTS.
2019-07-31 15:58:39 -07:00
Anders Kaseorg fd7803e7f4 settings: Unset STATIC_ROOT in development.
Django’s default FileSystemFinder disallows STATICFILES_DIRS from
containing STATIC_ROOT (by raising an ImproperlyConfigured exception),
because STATIC_ROOT is supposed to be the result of collecting all the
static files in the project, not one of the potentially many sources
of static files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Mateusz Mandera 34b9a7bcad populate_db: Refactor and rename send_messages().
We rename the send_messages function to generate_and_messages, and
factor out the actual sending part of it into a separate function, which
now gets the name send_messages().
2019-07-22 15:56:17 -07:00
Mateusz Mandera e5138c38b2 populate_db: Add a batching option. 2019-07-22 15:55:28 -07:00
Mateusz Mandera d1bab42da6 populate_db: Send generated messages in one do_send_messages call.
Sending messages one-by-one is significantly slower compared to taking
advantage of the batch-handling code in do_send_messages, so we pass all
the messages to the function in one go. This is memory-hungry if there's
a lot of messages, so we will allow splitting into smaller batches in
the next commit.
2019-07-22 15:55:28 -07:00
Mateusz Mandera a50524762c populate_db: Refactor message.pub_date generating code.
The code generating pub_dates for messages would fail to distribute them
across days if tot_messages was too large.
We refactor this code as a separate function (for clarity and to unit
test for the bug we're fixing), and change the structure and naming to a
form that more clearly describes what's happening. We also shift away
from the approach of all the float-to-int conversions as this is in
general tricky and bug prone - django's timedelta() handles floats as
arguments, so we take advantage of that.
2019-07-22 15:55:28 -07:00
Wyatt Hoodes 9b69c1f7fe Merge pull request #12833 from whoodes/clean-up-new-user-bot
bots: Replace remnants of new-user-bot with default-bot@zulip.com.
2019-07-20 14:38:05 -07:00
Mateusz Mandera c1a8ba2974 populate_db: Use do_change_is_admin for setting api_super_user. 2019-07-15 12:32:29 -07:00
Yashashvi Dave d7ee2aced1 models: Add `external_account` in custom profile field types.
Add new custom profile field type, External account.
External account field links user's social media
profile with account. e.g. GitHub, Twitter, etc.

Fixes part of #12302
2019-07-09 17:21:54 -07:00
Yashashvi Dave f813fdcd5e populate_db: Rename URL type custom profile field.
Rename URL type custom profile field in populate db to avoid confusion
with the "GitHub profile" custom external account profile field we'll
be adding shortly.
2019-07-09 17:21:39 -07:00
Vaibhav fa85b92eda populate_db: Add MD content to stream descriptions and user bio.
This adds a 'bold' element to `all` stream and 'code' element to
`test` stream in populate_db. Also changes the bio for Hamlet to
include a list.
2019-06-24 22:10:24 -07:00
Wyatt Hoodes fb58b54293 test-backend: Move `test_messages.json` to `var/<uuid>/test-backend`. 2019-06-24 12:23:09 -07:00
Tim Abbott bcc6949461 zilencer: Add better error handling for IntegrityError.
This provides a clean warning and 40x error, rather than a 500, for
this corner case which is very likely user error.

The test here is awkward because we have to work around
https://github.com/zulip/zulip/issues/12362.
2019-05-20 17:53:43 -07:00
neiljp (Neil Pilgrim) f3ef2a186c mypy: Enable strict-optional for calculate_first_visible_message_id. 2019-05-01 10:49:24 -07:00
Vishnu Ks ecdd3bea43 billing: Add cron job to run invoice_plans once a day.
Fixes #11960
2019-04-29 11:23:17 -07:00
Anders Kaseorg 9a9de156c3 lint: Fix calls to _() on computed strings.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:23:03 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Tim Abbott cfd5054f2d zilencer: Add indexes for remotecount models.
These new indexes improve the performance of checking whether our
Remote*Count models are up to date by a factor of ~1000.
2019-04-23 13:52:02 -07:00
Rishi Gupta e8741c448d refactoring: Move set_default_streams into do_create_realm.
After the commits leading up to this, the only meaningful use of this
function was in the realm creation process.
2019-03-21 12:33:19 -07:00
Rishi Gupta e71a1a2b4e onboarding: Remove initial streams other than general and core team.
The hope is that by having a shorter list of initial streams, it'll
avoid some potential confusion confusion about the value of topics.
At the very least, having 5 streams each with 1 topic was not a good
way to introduce Zulip.

This commit minimizes changes to the message content in
`send_initial_realm_messages` to keep the diff readable. Future commits will
reshape the content.
2019-03-21 12:30:14 -07:00
Tim Abbott 12756b48a0 populate_db: Fix help text for --test-suite. 2019-02-20 17:42:30 -08:00
Eeshan Garg 179b747769 streams: Refactor multi-option helpers into separate functions.
For internal stream messages, most of the time, we have access to
a Stream object. For the few corner cases where we don't, it is a
much cleaner approach to have a separate function that accepts a
stream name than having one multi-option helper that accepts both
names and objects.
2019-02-12 11:10:26 -08:00
Greg Price 49fd2e65de push notif: Add GCM options to bouncer API; empty for now.
The first use case for this will be setting `priority`,
coming up shortly.
2019-02-08 09:40:43 -08:00
Eeshan Garg c240008edb internal_prep_stream_message: Support accepting a Stream object.
If the caller has access to a Stream object, it is wasteful to
query a database for a stream by ID or name. In addition, not
having to go through stream names eliminates various classes of
possible bugs involved with getting a Stream object back.
2019-02-08 08:53:16 -08:00
Anders Kaseorg f5197518a9 analytics/zilencer/zproject: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:31:45 -08:00
Anders Kaseorg 4e21cc0152 views: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:23:43 -08:00
Tim Abbott 55ead5b77f zilencer: Fix buggy validation of installation_counts upload.
This was a simple copy-paste error.  It's probably worth a bit more
work on code duplication in this code path.
2019-02-02 11:51:22 -08:00
Tim Abbott 022c8beaf5 analytics: Add APIs for submitting analytics to another server.
This adds a new API for sending basic analytics data (number of users,
number of messages sent) from a Zulip server to the Zulip Cloud
central analytics database, which will make it possible for servers to
elect to have their usage numbers counted in published stats on the
size of the Zulip ecosystem.
2019-02-01 22:03:52 -08:00
Anders Kaseorg 79d888223f profiling: Use mkstemp for profile.data filename.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:08:48 -08:00
Tim Abbott 3c70a61132 models: Deduplicate definitions of push tokens.
Since c57c4cf703, these are no longer
different implementations, so we can deduplicate them.  Because Django
is smart, no migration is required.
2018-12-18 13:05:23 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Vishnu Ks 2889de6d90 populate_db: Extract subscribe_users_to_streams function. 2018-11-29 12:15:58 -08:00
Tim Abbott 9c224ccdd3 populate_db: Suppress logging output for push notifications.
This fixes a lot of spammy output of the form:

2018-11-27 17:46:48.279 INFO [zerver.lib.push_notifications] Sending push notification to user 46

when running populate_db, which is both confusing (since we're not
actually sending push notifications here) and spammy.
2018-11-27 09:50:17 -08:00
Tim Abbott e8290ccd6d add_new_user: Fix mention of running "python manage.py".
This won't work on Python 3.
2018-11-06 13:28:18 -08:00
Tim Abbott 68ab71eb8b push: Fix exceptions when removing push notifications.
Now that we allow multiple users to have registered the same token, we
need to configure calls to unregister tokens to only query the
targeted user_id.

We conveniently were already passing the `user_id` into the push
notification bouncer for the remove API, so no migration for older
Zulip servers is required.
2018-10-12 11:19:23 -07:00
Vishnu Ks d8c19cb003 models: Move billing models from zilencer to corporate. 2018-10-11 14:54:29 -07:00
Tim Abbott c57c4cf703 notifications: Fix push notifications with multiple realms.
Previously, Zulip did not correctly handle the case of a mobile device
being registered with a push device token being registered for
multiple accounts on the same server (which is a common case on
zulipchat.com).  This was because our database `unique` and
`unique_together` indexes incorrectly enforced the token being unique
on a given server, rather than unique for a given user_id.

We fix this gap, and at the same time remove unnecessary (and
incorrectly racey) logic deleting and recreating the tokens in the
appropriate tables.

There's still an open mobile app bug causing repeated re-registrations
in a loop, but this should fix the fact that the relevant mobile bug
causes the server to 500.

Follow-up work that may be of value includes:
* Removing `ios_app_id`, which may not have much purpose.
* Renaming `last_updated` to `data_created`, since that's what it is now.

But none of those are critical to solving the actual bug here.

Fixes #8841.
2018-10-10 16:15:52 -07:00
Vishnu Ks 59865797ac billing: Move billing tests from zilencer to corporate. 2018-10-01 15:44:02 -07:00
Vishnu Ks c01ebaa8d8 billing: Move billing management commands from zilencer to corporate. 2018-10-01 15:43:55 -07:00
Vishnu Ks 93b398bc0a billing: Move zilencer/lib/stripe to corporate. 2018-10-01 15:43:44 -07:00
Vishnu Ks 6914ee126c billing: Move billing-related views and urls to corporate. 2018-10-01 15:43:21 -07:00
Vishnu Ks 5a6b2ebb1f billing: Add ability for users to change their card. 2018-09-24 19:25:26 -07:00
Vishnu Ks 3e86fa84ea billing: Rename RealmAuditLog.STRIPE_CARD_ADDED to STRIPE_CARD_CHANGED. 2018-09-11 00:08:38 -07:00
Rishi Gupta 18b86a9499 billing: Fix RealmAuditLog accounting for downgrade. 2018-09-01 09:42:10 -07:00
Rishi Gupta 31ed4492ce billing: Add backend for downgrading. 2018-08-31 17:49:34 -07:00
Rishi Gupta b7c326a161 billing: Include account balance in prorated credits calculation. 2018-08-31 17:49:34 -07:00
Rishi Gupta bebd93b571 billing: Fix idempotency bug in setup_stripe management command. 2018-08-31 17:49:34 -07:00
Rishi Gupta 24b22e7e75 billing: Guard billing by BILLING_ENABLED rather than DEVELOPMENT. 2018-08-31 17:49:34 -07:00
Vishnu Ks 7b307fa58f billing: Set max_invites to 3000 for Premium organizations.
This is meant to be effectively infinite for legitimate use.
2018-08-28 16:03:34 -07:00
Vishnu Ks 98dac0f0a9 billing: Remove Customer.billing_user. 2018-08-27 12:35:46 -07:00
Vishnu Ks dcb7b15069 billing: Use UserProfile.is_billing_admin instead of Customer.billing_user. 2018-08-27 12:35:46 -07:00
Rishi Gupta eec07072ee billing: Add coupons. 2018-08-24 17:27:26 -07:00
Rishi Gupta d229948a43 billing: Make source optional when creating stripe customer. 2018-08-24 16:40:34 -07:00
Rishi Gupta 4ad8f20c28 billing: Add comment explaining race condition in customer creation. 2018-08-24 16:40:34 -07:00
Rishi Gupta 90407537a0 billing: Add comment explaining race conditions in initial subscription. 2018-08-22 21:30:48 -07:00
Rishi Gupta 16a5540e58 billing: Add comment to downgrading section in views.billing_home. 2018-08-22 19:20:06 -07:00
Rishi Gupta d081cf7b78 billing: Add transaction.atomic() for Customer creation and logging. 2018-08-22 19:20:06 -07:00
Rishi Gupta 3cf9cd00d6 billing: Remove idempotency key from upgrade process.
This would be relevant if we retried on network failure, but we don't.
2018-08-22 19:18:32 -07:00
Vishnu Ks 9bb338be11 models: Add plan_type to Realm. 2018-08-21 12:39:06 -07:00
Rishi Gupta c3a912f8b6 management commands: Add sleep_forever to lib/management. 2018-08-17 09:20:51 -07:00
Rishi Gupta 4181d575b1 billing: Move Stripe call to end in do_subscribe_customer_to_plan.
In general, everything that could throw an error and that could come before
the Stripe call should do so.
2018-08-13 20:24:08 -07:00
Rishi Gupta 56d4034db4 billing: Require successful charge to establish billing relationship.
There are several situations in which we want to create a Customer and
stripe.Customer object before we really have a billing relationship with a
customer. The main one is giving non-profit or educational discounts.
2018-08-13 20:24:08 -07:00
Rishi Gupta 27d4b1a5f0 billing: Add RealmAuditLogProcessor. 2018-08-13 17:12:50 -07:00
Rishi Gupta e804c563a8 billing: Update stripe error handling code. 2018-08-13 17:12:50 -07:00
Rishi Gupta d0e377428a billing: Fix typo in CONTACT_SUPPORT error message. 2018-08-10 22:48:10 -07:00
Rishi Gupta 2855864f54 billing: Rename RealmAuditLog.REALM_PLAN_QUANTITY_RESET. 2018-08-10 15:54:31 -07:00
Rishi Gupta abc044e869 billing: Rename RealmAuditLog.REALM_PLAN_STARTED. 2018-08-10 15:54:31 -07:00
Rishi Gupta d33d7e1c28 billing: Rename RealmAuditLog.REALM_CARD_ADDED. 2018-08-10 15:54:30 -07:00
Rishi Gupta bdb6d8213f billing: Rename RealmAuditLog.REALM_STRIPE_INITIALIZED. 2018-08-10 15:46:55 -07:00
Vishnu Ks 0c84260faa billing: Enforce full coverage in zilencer/tests. 2018-08-10 13:52:35 -07:00
Vishnu Ks 4895637a5a billing: Fix the BillingError description check in test.
The self.assertEqual statement won't get executed because
BillingError would be raised and the execution would exit
with block.
2018-08-10 13:52:35 -07:00
Vishnu Ks d32051b106 billing: Test that salt and seat_count are set correctly in /upgrade. 2018-08-10 13:52:35 -07:00
Vishnu Ks 132480c031 billing: Pass email to stripe.Customer.create. 2018-08-10 13:52:35 -07:00
Vishnu Ks d64ba5d2be billing: Update the charged amount when user changes plan in upgrade page. 2018-08-10 13:52:35 -07:00
Yago González 6a192ac84c utils: Move random API key generator as generate_api_key.
random_api_key, the function we use to generate random tokens for API
keys, has been moved to zerver/lib/utils.py because it's used in more
parts of the codebase (apart from user creation), and having it in
zerver/lib/create_user.py was prone to cyclic dependencies.

The function has also been renamed to generate_api_key to have an
imperative name, that makes clearer what it does.
2018-08-08 16:45:25 -07:00
Yago González f6219745de users: Get all API keys via wrapper method.
Now reading API keys from a user is done with the get_api_key wrapper
method, rather than directly fetching it from the user object.

Also, every place where an action should be done for each API key is now
using get_all_api_keys. This method returns for the moment a single-item
list, containing the specified user's API key.

This commit is the first step towards allowing users have multiple API
keys.
2018-08-08 16:35:17 -07:00
Rishi Gupta 2cdcf4b88f billing: Rename get_upcoming_invoice. 2018-08-06 16:24:09 -04:00
Rishi Gupta 89ca1d0d97 billing: Rename get_stripe_customer. 2018-08-06 16:24:09 -04:00
Rishi Gupta 086b7d76e5 billing: Rename customer_with_active_subscription in fixture generation.
Fixes regression introduced in 79dddd5.
2018-08-06 16:24:09 -04:00
Rishi Gupta 9f2b8a4a11 billing: Move checks from process_initial_upgrade into separate function. 2018-08-06 11:05:33 -04:00
Rishi Gupta 5719633992 billing: Update and unify billing error handling. 2018-08-06 11:05:33 -04:00
Rishi Gupta 79dddd5b80 billing: Rename customer_with_active_subscriptiion. 2018-08-06 11:05:33 -04:00
Vishnu Ks 18a9e4fc9c billing: Call mock directly in test_subscribe_customer_to_second_plan. 2018-08-06 11:05:33 -04:00
Vishnu Ks e06957bef5 billing: Raise exceptions instead of returning errors in upgrade flow. 2018-08-06 11:05:33 -04:00
Vishnu Ks 8643c681f0 billing: Separate initial upgrade logic to another function. 2018-08-06 11:05:33 -04:00
Vishnu Ks fa7b98030b billing: Add mock for cancel at period end subscriptions. 2018-08-04 19:17:34 -04:00
Vishnu Ks d89e07b8a7 billing: Add mock and test for extracting canceled subscription. 2018-08-04 19:17:34 -04:00
Vishnu Ks 84b148728f billing: Don't mock functions in test_extract_current_subscription. 2018-08-04 19:17:34 -04:00
Vishnu Ks c2023eaaa8 billing: Mention the exact type of stripe.Customers subscriptions attribute. 2018-08-04 19:17:34 -04:00
Vishnu Ks 036ad03b3a billing: Mention the exact return type for mock functions. 2018-08-04 19:17:34 -04:00
Vishnu Ks 8bcd4131fe billing: Rename retrieve_customer fixture. 2018-08-04 19:17:34 -04:00
Vishnu Ks b7d3a1a0f3 billing: Don't check for stripe key and plan objects in tests.
Stripe already returns an appropriate error in prod, and these checks are
just a hassle in tests.

Also fixes an error where the check for Plan.objects.exists() was missing
a "not".
2018-08-04 19:17:34 -04:00
Roman Godov c0806917ec models: Rename Realm.restricted_to_domain field.
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.

Fixes part of #10042.
2018-07-31 09:28:33 -07:00
Rishi Gupta 4554b564f7 billing: Rename RealmAuditLog.REALM_PLAN_QUANTITY_UPDATED. 2018-07-31 10:56:31 -04:00
Vishnu Ks abf485bfcb billing: Add mypy stubs to stripe imports.
Tweaked by tabbott to add an extra type: ignore.
2018-07-26 16:31:32 -07:00
Vishnu Ks d7b7430af0 billing: Add test for subscribing customer to second plan. 2018-07-25 23:03:55 -07:00
Vishnu Ks ba7db968d8 billing: Don't check for stripe config while running tests. 2018-07-25 23:03:55 -07:00
Vishnu Ks 51c82b6d4f models: Rename PLAN_QUANITY_UPDATED event type. 2018-07-25 23:03:55 -07:00
Vishnu Ks 975969aa89 stripe: Check for the existence of Pan objects in dev environment. 2018-07-25 23:03:55 -07:00
Vishnu Ks d6ed2c69be billing: Don't use self.realm and self.user in tests. 2018-07-25 23:03:55 -07:00
Vishnu Ks f98a86cdc2 billing: Don't show initial upgrade page in production for now. 2018-07-25 23:03:55 -07:00
Vishnu Ks b88613fd0b billing: Expand default_source object when retrieving customer. 2018-07-24 10:20:23 -07:00
Vishnu Ks 1225bfd6ee billing: Don't use data attribute for iterating through subscriptions.
Iterating through data attribute is same as iterating through
subscriptions.
2018-07-24 10:20:23 -07:00
Vishnu Ks 0bcc0ca0d8 stripe: Fix type of stripe_customer_id. 2018-07-24 10:20:23 -07:00
Vishnu Ks 15eff517f4 billing: Verify that plan names are not tampered. 2018-07-24 08:04:00 -07:00
Vishnu Ks 82fc82b7e2 billing: Sign and verify the seat count during upgrade. 2018-07-24 08:04:00 -07:00
Vishnu Ks d75054fb15 billing: Add function to sign strings. 2018-07-24 07:16:12 -07:00
Tim Abbott 888434e6cd test_stripe: Delete broken test.
This is an expedient way to fix master failing, which it has been
since f2861442f7.
2018-07-23 22:20:12 -07:00
Vishnu Ks eb48197f67 billing: Get upcoming invoice only if there is a live subscription. 2018-07-23 19:18:17 -07:00
Vishnu Ks f2861442f7 billing: Enforce Customer can have only one live subscription at a time. 2018-07-23 19:03:36 -07:00
Vishnu Ks db8700513b stripe: Fix type of stripe_plan_id in do_subscribe_customer_to_plan. 2018-07-23 17:55:15 -07:00
Vishnu Ks 28167585b1 models: Rename several RealmAuditlog event types to past tense.
These event types haven't been used in production yet, and thus are
save to change without a migration.
2018-07-22 20:00:05 -07:00
Max Nussenbaum 8fd900626d billing: Style billing page.
This adds styles to /billing.
2018-07-13 18:30:38 +05:30
Vishnu Ks 895c7703a6 billing: Show the billing page only to admins and billing users. 2018-07-12 12:51:38 +05:30
Vishnu Ks 377fac2864 billing: Pass stripe_customer instead of Customer in upgrade functions. 2018-07-12 12:51:38 +05:30
Vishnu Ks ff4c1ca2c1 models: Add SUBSCRIPTION_CREATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 89fefcd1a2 mypy: Replace Any with stripe types in zilencer/lib/stripe.py. 2018-07-09 15:24:57 +05:30
Shubham Dhama 7cbacdb8c2 populate_db: Integrate populate_analytics_db.
As detailed in the documentation changes, this simplifies the
development workflow for doing UI work on the /stats pages.

The cost is a ~10% increase the time it takes to run `populate_db`,
which doesn't happen very often (and for most purposes manifests as a
1% increase in the time it takes to rebuild the database from scratch).
2018-07-09 15:05:48 +05:30
Rishi Gupta b5753d0ddc billing: Add initial support for seat based plans.
The main remaining todo for correctly populating
RealmAuditLog.requires_billing_update is supporting the de-seating (and
corresponding re-seating) that happens after being offline for two weeks.
2018-07-09 14:33:08 +05:30
Rishi Gupta 16334a1ba7 billing: Update billing system. 2018-07-09 14:33:07 +05:30
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Yashashvi Dave 8909cb1d15 custom fields: Allow list of users in user type of custom fields.
Allow user to add more than one user-value in user type of custom
fields.

Tweaked by tabbott to improve the models.py code and type annotations.
2018-06-16 09:37:49 -07:00
Rishi Gupta 634c4516a7 portico: Add svgs of sea creatures. 2018-06-06 13:57:55 -07:00
Tim Abbott 27e7770540 populate_db: Initialize the outgoing webhook bot properly.
Previously, we created the bot, but didn't initialize its service
object.
2018-06-01 08:48:17 -07:00
Robert Hönig 497cd4a1f9 outgoing webhooks: Change default URL in all tests.
This reflects the changes to the default URL publicly
displayed to the user. It also changes the default
URL of the default test server outgoing webhook, which
prevented the test server flaskbotrc from working out
of the box.
2018-05-28 10:40:44 -07:00
Yashashvi Dave e82c879b85 custom fields: Add user type of custom fields.
Fixes #8878
2018-05-27 23:01:21 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Aditya Bansal 8cfb437677 provision: Don't run create_realm_internal_bots management command.
We essentially stop running create_realm_internal_bots during
every provisioing and move its operations to run from populate db.
In fact to speed things up a bit we actually make populate db call the
funcs which create_realm_internal_bots calls behind the scenes.

Fixes: #9467.
2018-05-23 13:34:52 +05:30
Tim Abbott 1b47b2e7cd populate_db: Fix an out-of-date mypy annotation. 2018-05-22 14:46:03 -07:00
Tim Abbott 8a64d8ef06 populate_db: Fix stream colors being generated wrong.
Previously, the stream colors index i was accidentally a function only
of the user, so each user got the same color for all their streams.

This should provide a lot nicer-looking development environment
experience.
2018-05-22 14:29:43 -07:00
Tim Abbott 4c8a69e3fb populate_db: Add an initial stream for the default guest user. 2018-05-22 14:29:43 -07:00
Sampriti Panda 4ce5a85a49 populate_db: Make test_suite data generation more deterministic. 2018-05-22 14:07:55 -07:00
Tim Abbott cec7686f3d push notifications: Clean up unregistered/bad APNS tokens.
We've had this sort of logic for GCM for a long time; it's worth
adding for APNS as well.

Writing this is a bit of a reminder that I'm not a fan of how our unit
tests for push notifications work.
2018-05-21 11:30:56 -07:00
Shubham Padia 4d70d032bc populate_db: Make is_web_public default to False.
Explicit mentions of `is_web_public` as False in the stream dict to be
passed to `bulk_create_streams` have been removed.
2018-05-16 13:40:22 -07:00
Shubham Padia 315047a38b populate_db: Make invite_only default to False.
Explicit mentions of `invite_only` as False in the stream dict to be
passed to `bulk_create_streams` have been removed.
2018-05-16 13:40:10 -07:00
Shubham Padia 7b4f6e0058 tests: Make is_announcement_only true for `announce` stream in populate_db.
Makes announce stream `is_announcement_only` for the dev db for easier
manual testing. The default value for `is_announcement_only` in
`bulk_create_streams` is False.
2018-05-16 13:35:45 -07:00
Aditya Bansal 67bf71472a zilencer: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Tim Abbott 7cbff8b521 push registration: Use standard error message for auth problems.
This avoids adding an unnecessary new translated string.
2018-05-04 09:04:39 -07:00
Tim Abbott 43098a6f7c zilencer: Add automated signup system for push notifications.
Based on an initial version by Rishi Gupta.

Fixes #7325.
2018-05-03 21:27:49 -07:00
Rishi Gupta d18102d1c6 zilencer: Remove uniqueness constraint on RemoteZulipServer.hostname.
Enforcing the unique constraint adds an unnecessary support burden for
figuring out who actually controls a given hostname, and in particular, for
verifying updates to the org id/key on a re-install of the Zulip server.
2018-05-03 12:50:29 -07:00
Rishi Gupta 42a99e8c1d zilencer: Rename remote_server_unregister_push. 2018-05-03 12:50:29 -07:00
Rishi Gupta 76650f5930 zilencer: Rename remote_server_register_push to register_remote_push_device. 2018-05-03 12:50:29 -07:00
Aditya Bansal 9629be689b populate_db: Add a web public stream to dev database.
We flip the Stream "Rome" to be a web public stream. Also we add
attribute is_web_public in various stream dicts and in the
bulk_create_streams function of bulk_create.py responsible for
default stream creation in dev environment.
2018-05-02 15:23:33 -07:00
Yashashvi Dave 0d7d94d0db custom fields: Add support for custom URL field type. 2018-04-30 10:53:23 -07:00
Yashashvi Dave 4033f210af custom fields: Add support for custom date field type. 2018-04-30 23:04:25 +05:30
Tim Abbott 8344bd171e populate_db: Properly initialize the email_gateway_bot.
Apparently, this bot account was not properly being tagged as an API
super user in the test database; resulting in incorrect behavior if we
tried to send to a private stream in a test.

(Note that there seems to also be a similar issue in production, that
we don't understand the cause of; that is unrelated).
2018-04-27 13:56:04 -07:00
Umair Khan cf2f6b38dd profile: Add choice field.
Fixes part of #8878
2018-04-26 00:35:51 -07:00
Shubham Dhama 26d2ffa821 populate_db: Add new user of "guest" type.
The purpose of this user is to act as a guest.
(This is a preliminary step in adding the guest type of user
and is a part of #8385.)
2018-04-20 16:20:00 -07:00
Preston Hansen 76d6c71595 tests: Move zerver/fixtures to zerver/tests/fixtures for clarity.
Fixes #9153.
2018-04-19 21:50:17 -07:00
Umair Khan 0420b89468 profile: Add hint attribute to custom profile fields.
This is the model for #8876.
2018-04-19 11:32:35 -07:00
Umair Khan c30a282dd9 profile: Remove integer and float fields. 2018-04-02 09:46:21 -07:00
sinwar 79672fffc6 populate_db: Pre-populate open graph cache to avoid network activity.
This prevents annoying network errors when trying to populate the
database without Internet access.

Cleaned up a bit by tabbott.

Fixes #8289.
2018-03-27 16:42:59 -07:00
Tim Abbott 0b3b363cd8 zilencer: Fix buggy RemotePushDeviceToken __str__.
Caught by mypy!
2018-03-23 11:30:50 -07:00
Tim Abbott 5e4b445649 zilencer: Add __str__ methods for Remote models. 2018-03-23 11:02:51 -07:00
Jack Weatherilt 3396cfc2ef refactoring: Replaced occurences of create_stream_if_needed.
Issue #2088 asked for a wrapper to be created for
`create_stream_if_needed` (called `ensure_stream`) for the 25 times that
`create_stream_if_needed` is called and ignores whether the stream was
created. This commit replaces relevant occurences of
`create_stream_if_needed` with `ensure_stream`, including imports.

The changes weren't significant enough to add any tests or do any
additional manual testing.

The refactoring intended to make the API easier to use in most cases.
The majority of uses of `create_stream_if_needed` ignored the second
parameter.

Fixes: #2088.
2018-03-21 16:47:36 -07:00
YJDave 6ac687790c populate_db: Remove the `test_suite` check for custom profile fields.
To ensure that we have some basic data for custom profile settings,
in the `populate_db` data set, remove `options['test_suite']` check
for adding intial custom profile data.
2018-03-21 16:05:31 -07:00
Harshit Bansal a49655e0d4 emoji: Migrate realm emoji to be addressed by `id` rather than `name`.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.

Fixes: #6977.
2018-03-20 22:24:44 -07:00
Eeshan Garg af56df7723 webhooks: Enable custom topics and default PM notifications.
This commit adds a generic function called check_send_webhook_message
that does the following:
* If a stream is specified in the webhook URL, it sends a stream
  message, otherwise sends a PM to the owner of the bot.
* In the case of a stream message, if a custom topic is specified
  in the webhook URL, it uses that topic as the subject of the
  stream message.

Also, note that we need not test this anywhere except for the
helloworld webhook. Since helloworld is our default example for
webhooks, it is here to stay and it made sense that tests for a
generic function such as check_send_webhook_message be tested
with an actual generic webhook!

Fixes #8607.
2018-03-16 11:34:20 -07:00
Tim Abbott 56b1239ad5 populate_db: Include some initial data for custom profile fields.
This is currently only for manual testing; creating these would break
the existing custom profile test suite.
2018-03-11 18:08:17 -07:00
Nick Al cab3c8c32c populate_db: Spread message timestamps over a few days.
This makes it easier to work on features that depend on messages
having been sent in the past (E.g. the date parts of recipient bars).

The new feature only works with --threads=1; since with the ~100
default messages that populate_db generates, the multi-threaded
feature shouldn't have significant performance impact (and it would be
tricky to make increasing timestamps work with the multi-threaded
model), it's reasonable to just set the default number of threads to 1
for now and have this timestamp-spreading feature only supported with
--threads=1.

Fixes #8277.
2018-02-24 08:11:27 -08:00
Greg Price 271cfd4d7a translation: Try the new `err_` marking on a handful of messages. 2018-02-15 13:38:09 -08:00
neiljp (Neil Pilgrim) bed56b253e mypy: Add clarifying variable in add_new_user.py. 2018-02-13 11:40:52 -08:00
Vishnu Ks 003cb0478a stripe: Use sources.list instead of sources.all.
Sources.all would be depricated soon in favor of sources.list.
See https://github.com/stripe/stripe-python/blob/
v1.77.1/stripe/api_resources/list_object.py#L18
2018-02-12 12:45:46 -08:00
Umair Khan 0d796e6c02 django-2.0: Add on_delete to OneToOneField. 2018-01-31 12:07:36 -08:00
Greg Price f9b12952f8 stripe: Move error handling into stripe.py too.
This completes the separation of our logic for managing Stripe
customers from the view code for the billing page.

As we add more features to our Customer model and to our Stripe
integration, we might further separate those two things; but for now
they're nearly synonymous and there's no problem in them being mixed
together.
2018-01-30 12:44:35 -08:00
Greg Price 0b81762350 billing: Move most Stripe code to its own file.
We'll handle the error-handling in a separate commit, as it's still
entangled with the view function.
2018-01-30 12:44:35 -08:00
Greg Price 5feb31a957 billing: Start pulling apart the page from the logic.
Pull the code that talks to Stripe out into its own functions.
In a followup commit we'll move these to a separate file, as well
as the error-handling logic that remains in the view function
for now.

Also fix the translation markings: the translated string must be a
constant (e.g. a format string), or else translation is impossible.

Viewing with `-b` shows the few changes that happen in the logic
as it moves out of the view function; viewing without shows the
few changes in the rest of the view function.
2018-01-30 12:44:35 -08:00
rht 8106a25e61 django-2.0: Add on_delete on ForeignKeys.
In Django 2.0, one must specify the on_delete behavior for all
ForeignKeys explicitly.
2018-01-30 10:53:54 -08:00
Brock Whittaker 3b600d5591 /billing/: Rename "payment.html" => "billing.html".
This matches the URL path /billing/ to the filename "billing.html".
2018-01-29 17:24:28 -08:00
Vishnu Ks a11b742984 messages: Calculate value of first visible message ID using cron job.
[greg: Fixed buggy time conversion in estimate_recent_messages.]
2018-01-24 17:15:08 -08:00
Vishnu Ks b762b839d4 stripe: Make newly added card default source. 2018-01-22 19:31:17 -08:00
Greg Price 1ef2d9d637 billing: Simplify logging of Stripe errors.
Several changes:
* De-duplicate code for different error types.
* No need to list lots of error subtypes where we aren't treating
  them differently; StripeError is the base class of them all.
* Unexpected, non-Stripe-related, exceptions we can handle in the normal
  way.  Just make them show up in the billing-specific log too.
* The Stripe client library already logs type, code, param, and message
  before raising an error, so we don't need to repeat those; just add the
  HTTP status code (because it's not there already and sure why not),
  and the Python exception type the client library chose to raise
  in case that makes things a bit easier to interpret.
2018-01-17 18:10:17 -08:00
Greg Price 858b9d7bc1 billing: Log important requests to Stripe. 2018-01-17 18:10:17 -08:00
Greg Price e8ceea7f0f billing: Send logs to a dedicated file.
This will help make them easy to read through.  Include messages to
the logger name `stripe`, which Stripe's client library logs to.
2018-01-17 18:10:17 -08:00
Greg Price 3aa5529b96 billing: Better error message for dev on lacking Stripe credentials.
Normal server admins will never run this code, and zulipchat.com will
have this information configured before users see it, so this message
is really just for development.
2018-01-17 16:44:04 -08:00
Greg Price 2b6e526ea2 zilencer: Make /billing appear without i18n prefix.
This copies what we do in `zproject/urls.py` for pages in the
main Django app.
2018-01-17 16:44:04 -08:00
Vishnu Ks 0bca0286a1 billing: Integrate Stripe, using Stripe Checkout.
Stripe Checkout means using JS code provided by Stripe to handle
almost all of the UI, which is great for us.

There are more features we should add to this page and changes we
should make, but this gives us an MVP.

[greg: expanded commit message; fixed import ordering and some types.]
2018-01-17 16:43:54 -08:00
Vishnu Ks a978336765 billing: Create model for customers paying with Stripe.
[greg: reduced migration dependencies, revised commit message.]
2018-01-17 16:15:14 -08:00
Greg Price 9a4cd54a25 zilencer: Add a simple management command to create a RemoteZulipServer.
This saves us from having to go into a `manage.py shell` to do this,
and adds a bit more structure like the usage message.
2018-01-08 17:46:46 -08:00
Marco Burstein 6f8792729d manage.py: Move the maximum digits calculation to outside of a loop. 2018-01-02 17:21:28 -05:00
Marco Burstein ff5a73e027 manage.py: Make it easy to create lots of streams with `populate_db`.
Fix #7968.
2018-01-02 16:48:03 -05:00
Viraat Chandra 65337603b6 mypy: Use Python 3 syntax for typing in `zilencer/views.py`. 2017-12-26 08:31:43 -05:00
rht 92d62f62e1 zilencer/management: Use Python 3 syntax for typing. 2017-12-12 17:41:12 -08:00
Greg Price 50e451cd7b populate_db: Create an email-reusing Cordelia in the "lear" realm. 2017-11-28 16:23:10 -08:00
Greg Price a753f49e6a populate_db: Add a "lear" realm for testing cross-realm email sharing.
We don't yet actually allow sharing emails across realms, but we
will soon.  This lets us start testing some pieces of those changes.
2017-11-28 16:23:10 -08:00
rht 138c486548 zilencer: Use python 3 syntax for typing. 2017-11-22 12:29:03 -08:00
Rishi Gupta 42652713c8 onboarding: Refactor away setup_initial_private_stream.
It is a confusing name, now that the stream creation part has been taken
away.
2017-11-21 17:39:50 -08:00
Tim Abbott 708872f361 populate_db: Use call_consume_in_tests when generating test data.
We'd rather this work be just executed immediately, rather than
queued, since queued events can confuse the queue workers if the
database is dropped and recreated repeatedly.
2017-11-21 09:43:54 -08:00
Tim Abbott 64230b3bd5 populate_db: Remove the 'simple' realm.
The 'simple' realm was super broken and confusing for new users.  We
should replace this with having an easy way to make a new realm in
development, done properly.

Fixes #6116.
2017-11-15 22:42:05 -08:00
Tim Abbott 2b43a0302a python: Sort imports in smaller apps. 2017-11-15 15:55:49 -08:00
Umair Khan ad8f5650a3 user-groups: Generate test data of user groups. 2017-11-13 13:08:15 +05:00
rht 548c96ad86 zilencer: Text-wrap long lines exceeding 110. 2017-11-07 17:24:06 -08:00
rht 995b300840 zilencer: Remove unused imports (F401). 2017-11-07 16:37:01 -08:00
rht ccf2792c1c refactor: Remove six.moves.configparser import. 2017-11-07 10:51:44 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
rht 0e71b243ce zilencer: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
neiljp (Neil Pilgrim) 6c995db231 mypy: Use Iterator[Any] as return type of queryset_iterator. 2017-11-04 19:47:45 -07:00
Harshit Bansal 233595b2f7 actions.py: Rename `do_add_reaction()` to `do_add_reaction_legacy()`. 2017-10-31 15:42:35 -07:00
Tim Abbott 1cd017288d views: Fix imports of REQ/has_request_variables from the wrong place.
These were never in zerver/decorator.py, and so it makes sense to
import them zerver/lib/request.py, mostly for ease of finding things.
2017-10-27 15:07:31 -07:00
Vishnu Ks 1d94119d31 actions: Call send_initial_pms from process_new_human_user. 2017-10-25 14:14:59 -07:00
Tim Abbott 735b49e505 zilencer: Fix uniqueness rules for RemotePushDeviceToken.
Since a user could use the same installation of the Zulip mobile app
with multiple Zulip servers, correct behavior is to allow reusing the
same token with multiple Zulip servers in the RemotePushDeviceToken
model.
2017-10-18 21:24:06 -07:00
rht 691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
rht 2f3ae84e5a py3: Remove all `__future__ import division`. 2017-10-17 23:09:12 -07:00
rht b2ad8fd747 py3: Remove all `from __future__ import unicode_literals`.
This was mostly used in migrations, so it's a pretty safe change.
2017-10-17 23:07:42 -07:00
Tim Abbott 66a3f514d4 zilencer: Remove long-obsolete Deployment model.
This was a precursor to RemoteZulipServer, which is no longer used for
anything, and was cluttering the codebase.
2017-10-11 23:48:01 -07:00
Tim Abbott 51006f4a43 populate_db: Create bots before sending messages.
Apparently, the order we'd just created in
cbbdd539f4 broke populate_db; this is a
better and more sustainable fix to the overall problem.
2017-10-11 23:27:00 -07:00
Tim Abbott cbbdd539f4 populate_db: Fix creating welcome-bot at the wrong place.
While it might be useful to have created welcome-bot earlier in a
certain sense, it's definitely not a good idea in this populate_db
implementation, because doing so threw off the random initial
assignment of users to streams and thus broke the casper tests.
2017-10-11 22:17:09 -07:00
derAnfaenger 5ddc336844 tests: Add welcome bot as user. 2017-10-11 20:45:42 -07:00
Brock Whittaker 835b6c7059 components: Make checkboxes darker.
This makes the standard checkboxes 7% darker and makes the disabled
ones about 12% darker + 7% darker than they were before, to
increase visibility.

Fixes: #6331.
2017-10-11 13:15:44 -07:00
rht 8d289b8228 zilencer: Remove absolute_import. 2017-09-27 20:20:07 -07:00
rht ed9b3ca440 zilencer: Remove import print_function. 2017-09-27 18:06:47 -07:00
Robert Hönig c77b245944 backend tests: Add 'AARON' as bot owner of OUTGOING_WEBHOOK_BOT. 2017-09-26 16:11:27 -07:00
Rishi Gupta 0727f045f5 zilencer: Add management commands to create new dev users and realms. 2017-09-22 06:35:43 -07:00
Greg Price 613d093d7d push notifs: Implement APNs with new API.
And it works!

A couple of things still to do:

 * When a device token is no longer active, we'll get HTTP status 410.
   We should then remove the token from the database so we don't keep
   trying to push to it.  This is fairly urgent.

 * The library we're using has a nice asynchronous API, but this
   version doesn't use it.  This is OK now, but async will be
   essential at scale.
2017-08-26 14:16:05 -07:00
Rishi Gupta b32fe9f2a0 features: Update screenshot and add tool to generate.
This adds a new management command that should make generating or
updating screenshots a bit easier in the future.
2017-08-22 11:32:22 -07:00
Tim Abbott d7f8781b06 populate_db: Remove unnecessary print statement. 2017-08-15 12:55:56 -07:00
Tim Abbott 2858c13513 populate_db: Use caching helpers to get users. 2017-08-15 12:55:00 -07:00
Tim Abbott 27b79e7189 populate_db: Fix duplicate test data when run multithreaded.
The previous implementation resulted in each of the 10 threads using
the same random seed and thus same messages.

Fixes #6028.
2017-08-05 11:22:11 -07:00
Darren Fix 69767a43cb Create complex sample messages for dev VM.
Create a generator script to pull lines from a play, enhancing
random lines with emoji, Markdown and other flair.

With numerous contributions from Rein Zustand and Tim Abbott to finish
the project.

Fixes: #1666.
2017-08-01 20:54:39 -07:00
Aditya Bansal 9e372c465c populate_db: Set pointer as message_id of last user sent message. 2017-07-28 11:57:52 -07:00
Harshit Bansal eeb25b6b4b populate_db: Add a test realm emoji while populating databases.
Add `green_tick` realm emoji which can be used while performing tests.
2017-07-27 18:57:06 +00:00
Vishnu Ks 7eaec543f1 zilencer: Use get_user in populate_db.py. 2017-07-18 17:14:05 -07:00
Vishnu Ks df664db7b4 zilencer: Remove unused imports from populate_db.py. 2017-07-18 17:14:05 -07:00
Vishnu Ks 9a44794d14 zilencer: Eliminate get_user_profile_by_email from sync_api_key. 2017-07-18 17:14:05 -07:00
Vishnu Ks b0b809b195 zilencer: Eliminate get_user_profile_by_email from print_initial_password. 2017-07-18 17:14:05 -07:00
Vishnu Ks 754db8ec9f zilencer: Eliminate get_user_profile_by_email from profile_request. 2017-07-18 17:14:05 -07:00
Aditya Bansal f2d6194ae1 actions: Start logging subscription activities in RealmAuditLog. 2017-07-17 17:23:41 -07:00
Aditya Bansal 0c4165a5bd pep8: Add compliance with rule E261 to populate_db.py. 2017-07-11 11:56:50 -07:00
Vaida d5517bae36 Delete the old zulip.com "referrals" system.
This system hasn't been in active use for several years, and had some 
problems with it's design.  So it makes sense to just remove it to declutter
the codebase.

Fixes #5655.
2017-07-07 14:59:18 -07:00
Tim Abbott d6e0960ca2 push_notifications: Fix mypy annotation logic around push tokens.
I'm not 100% confident this is long-term correct, but at least it's
consistent.
2017-07-07 10:54:37 -07:00
Tim Abbott 0606ba88df push_notifications: Verify that token types are valid.
We only have two types of push notification tokens, so we should
validate that in the bouncer code path.
2017-07-07 10:28:36 -07:00
Tim Abbott 40425044c4 push_notifications: Validate APNS token format in bouncer.
This prevents a buggy old Zulip server from forwarding invalid-format
push notification tokens to the push notification bouncer service.

As part of this change, we switch the token from Text to str to match
the rest of the code path.
2017-07-07 10:28:36 -07:00
Rishi Gupta 36cd122905 models: Change default org_type to CORPORATE.
Once we implement org_type-specific features, it'll be easy to change a
corporate realm to a community realm, but hard to go the other way. The main
difference (the main thing that makes migrating from a community realm to a
corporate realm hard) is that you'd have to make everyone sign another terms
of service.
2017-06-29 15:14:58 -07:00
Umair Khan 323fbdefa8 zilencer: Add on_delete in foreign keys.
on_delete will be a required arg for ForeignKey in Django 2.0. Set it
to models.CASCADE on models and in existing migrations if you want to
maintain the current default behavior.
See https://docs.djangoproject.com/en/1.11/ref/models/fields/#django.db.models.ForeignKey.on_delete
2017-06-13 15:14:00 -07:00
Aditya Bansal bcc247a5b1 pep8: Add compliance with rule E261 zilencer/urls.py. 2017-05-31 17:07:15 -07:00
vaibhav 9cf9837f12 webhooks: Add outgoing webhook bot user to development database. 2017-05-29 16:01:23 -07:00
Christian Hudon c80e6edb4e mypy: Declare models with null=True Optional. 2017-05-23 14:36:40 -07:00
Tim Abbott 0b2388bda9 push_notifications: Remove DeviceTokenType logic.
The syntax wasn't valid on Python 3.5, and the new code is somewhat
more readable anyway.
2017-05-16 12:26:55 -07:00
Umair Khan c4e5b75ead push_notification: Push data from Zilencer. 2017-05-16 12:26:55 -07:00
Umair Khan 4a864c7515 push_notification: Send data to notification bouncer. 2017-05-16 12:26:55 -07:00
Tim Abbott c7d2caab6e zilencer: Add auto_now to RemoteZulipServer.last_updated.
This should simplify the logic needed to create one of these.
2017-05-15 17:04:33 -07:00
Umair Khan e649c05aed views.py: Create validator for remote server. 2017-05-11 12:04:16 -07:00
Aditya Bansal bbccd8ab28 pep8: Add compliance with rule E261 to zilencer/models.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal b07273c2d0 pep8: Add compliance with rule E261 to commands/profile_request.py. 2017-05-07 23:21:50 -07:00
Tim Abbott 60225012e6 unread: Add tool for marking all messages as unread for testing.
This tool can save a lot of manual work in testing our unread counts
logic.
2017-04-25 15:40:12 -07:00
Tim Abbott cddee49e75 Add support infrastructure for push notification bouncer service.
This is an incomplete cleaned-up continuation of Lisa Neigut's push
notification bouncer work.  It supports registration and
deregistration of individual push tokens with a central push
notification bouncer server.

It still is missing a few things before we can complete this effort:
* A registration form for server admins to configure their server for
  this service, with tests.
* Code (and tests) for actually bouncing the notifications.
2017-04-18 23:03:06 -07:00
Tim Abbott 8ae052a9d8 populate_db: Set a description for the default realm. 2017-04-17 21:59:21 -07:00
hackerkid b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00
Harshit Bansal ac2172e233 models: Rename RealmAlias model to RealmDomain.
Includes a migration.
2017-04-04 15:48:03 -07:00
Rishi Gupta 8617c2d594 populate_db: Make Iago staff.
Will make it easier to test the /activity page.
2017-03-27 16:43:35 -07:00
Rishi Gupta 30024d0a8f models: Remove Realm.domain. 2017-03-25 19:55:48 -07:00
Tim Abbott 3617ebfd41 api: Rename get_old_messages to get_messages in the backend.
Fixes #1315.
2017-03-23 23:52:44 -07:00
Rishi Gupta ba228b75b1 zilencer: Use Realm.string_id for get_deployment_from_domain.
This is dead code. Getting rid of realm__domain since we're about to remove
that field from Realm.
2017-03-14 17:17:42 -07:00
Rishi Gupta 871c754369 bulk_create: Remove unused function bulk_create_realms. 2017-03-13 14:42:55 -07:00
Tim Abbott 950e06cf32 populate_db: Remove unused Deployment import. 2017-03-06 00:11:53 -08:00
Tim Abbott 10a52147fc zilencer: Remove obsolete create_deployment command. 2017-03-06 00:10:03 -08:00
Tim Abbott e1c1f96f9e zilencer: Eliminate submit_feedback indirection. 2017-03-06 00:01:57 -08:00
Tim Abbott ed5b76f566 zilencer: Move feedback code to zerver/lib/feedback. 2017-03-06 00:01:57 -08:00
Tim Abbott 1f9d93bc96 submit_feedback: remove unused domainish variable. 2017-03-06 00:01:57 -08:00
Tim Abbott 69061e6db4 zilencer: Remove obsolete lookup_endpoints_for_user endpoint.
This endpoint was part of the same obsolete system as
desktop_sso_dispatch.
2017-03-06 00:01:56 -08:00
Tim Abbott 9e8023843a zilencer: Remove obsolete desktop_sso_dispatch.
This feature has been obsolete since when Zulip was released as open
source software, since it's purpose was to avoid putting a "server
url" prompt in the desktop app, and now that prompt is required
anyway.
2017-03-06 00:01:47 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 1453a5bfda Change string_id of test zephyr realm from mit to zephyr.
Also changes Realm.is_zephyr_mirror_realm to use string_id=zephyr instead of
domain=mit.edu.
Part of a larger migration away from Realm.domain.
2017-03-04 12:18:01 -08:00
Rishi Gupta 15d60fa7ed Change now() to timezone.now() throughout codebase.
Change `from django.utils.timezone import now` to
`from django.utils import timezone`.

This is both because now() is ambiguous (could be datetime.datetime.now),
and more importantly to make it easier to write a lint rule against
datetime.datetime.now().
2017-02-28 16:03:37 -08:00
Tim Abbott 8691f1466a populate_db: Disable use of memcached when populating database.
This fixes an issue where one would get errors of the form:

`ValueError: unsupported pickle protocol: 3`

in a `run-dev.py` server run against Python 2 if you ran `provision`.
Provision currently runs `populate_db` with Python 3, storing Python 3
based data in memcached, which then can't be read by Python 2.
2017-02-26 21:08:47 -08:00
Tim Abbott 08a7e0db44 render_messages: Fix buggy handling of edit history. 2017-02-18 22:33:26 -08:00
Tim Abbott 080182ad17 populate_db: Fix broken initial presence data.
Apparently, the initial data just set everyone to "API" in a broken way.
2017-02-10 23:52:32 -08:00
Tim Abbott 7e8d3d9a46 populate_db: Add --extra-bots option to create extra bots. 2017-02-10 17:40:45 -08:00
Steve Howell 4be2d6577d Add a new community realm to our test databases.
The realm with string_id of "simple" just has three users
named alice, bob, and cindy for now.  It is useful for testing
scenarios where realms don't have special zulip.com exception
handling.
2017-01-26 19:25:34 -08:00
Tim Abbott dd9e0b8463 errors: Move do_report_error into zerver/lib/. 2017-01-23 23:18:28 -08:00
Tim Abbott e6fc4ae27d errors: Move zilencer/error_notify.py to zerver/lib/. 2017-01-23 23:17:09 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott d6e38e2a5c lint: Clean up E123 PEP-8 rule. 2017-01-23 21:34:26 -08:00
Tim Abbott e9158dd520 lint: Clean up E121 PEP-8 rule. 2017-01-23 21:02:39 -08:00
Tim Abbott c0c9dfb66d populate_db: Fix crash by removing do_send_message.
This old helper has for years been used only by populate_db, and got
buggy (as of a recent refactoring).  So we just call do_send_messages
directly instead.

Fixes the provisioning error we currently get in Travis CI.
2017-01-21 22:07:36 -08:00
Tim Abbott 99c5563bc6 internal_send_message: Make realm argument mandatory.
A lot of care has been taken to ensure we're using the realm that the
message is being sent into, not the realm of the sender, to correctly
handle the logic for cross-realm bot users such as the notifications
bot.
2017-01-21 21:37:30 -08:00
Tommy Ip 3823376b3f zilencer: Fix bare except clause. 2017-01-08 16:22:21 -08:00
Rishi Gupta 2b0a7fd0ba Rename models.get_realm_by_string_id to get_realm.
Finishes the refactoring started in c1bbd8d. The goal of the refactoring is
to change the argument to get_realm from a Realm.domain to a
Realm.string_id. The steps were

* Add a new function, get_realm_by_string_id.

* Change all calls to get_realm to use get_realm_by_string_id instead.

* Remove get_realm.

* (This commit) Rename get_realm_by_string_id to get_realm.

Part of a larger migration to remove the Realm.domain field entirely.
2017-01-04 17:12:23 -08:00
Rafid Aslam 50afe5a6c2 sync_api_key: Fix `ConfigParser` deprecation warnings.
Fix `ConfigParser` deprecation warnings in
`zilencer/management/commands/sync_api_key.py`.

Fixes #2727.
2016-12-30 11:52:22 -08:00
Rafid Aslam aece2c7c45 populate_db: Fix naive datetime warnings
Fix naive datetime warnings in
`zilencer/management/commands/populate_db.py`.

Fixes #2729.
2016-12-20 17:48:22 -08:00
Tim Abbott d3b1f6e273 populate_db: Add notifications stream for Zulip realm. 2016-12-15 22:43:08 -08:00
Rishi Gupta 6b39f3222c Remove Realm.domain from populate_db.py.
Fixes #2286.
2016-12-15 13:22:17 -08:00
Rishi Gupta fc188de8e0 bulk_create: Refactor bulk_create_users to take Realm instead of domains.
Previously bulk_create_users would figure out a user's realm from their
email domain. Now require that a realm be passed explicitly.
2016-12-15 13:22:17 -08:00
Rishi Gupta 56c8b58cbc populate_db: Remove create_streams.
Is extraneous indirection.
2016-12-15 13:22:17 -08:00
Rishi Gupta ee9b59321c populate_db: Refactor create_streams.
Remove unused field from stream_dict.
2016-12-15 13:22:17 -08:00
Rishi Gupta fbf48f7ec6 bulk_create: Refactor bulk_create_streams to not take domain dictionary.
First step in cleaning up populate_db.create_streams and
bulk_create.bulk_create_streams. Part of a series of commits to remove
Realm.domain from populate_db.
2016-12-15 13:22:17 -08:00
reyha 82e32ad255 Access realm by `string_id` in management commands.
`Realm.string_id` replaces 'Realm.domain'
in the management commands.

Fixes #2325.
2016-12-14 10:38:03 -08:00
Juan Verhook 9869fb8d87 Add UserPresence rows to test databases.
Fixes: #2426

mypy: Added annotations to populate_db.py
2016-12-09 15:41:21 -08:00
Joy Chen dd95a5e03f streams: Add default stream description population data. 2016-12-07 21:02:17 -08:00
anirudhjain75 beaa62cafa mypy: Convert several directories to use typing.Text.
Specifically, these directories are converted: [analytics/, scripts/,
tools/, zerver/management/, zilencer/, zproject/]
2016-12-07 20:51:05 -08:00
nikolay abc2ff4a06 pep8: Fix many rule E128 violations.
[Tweaked by tabbott to adjust some approaches used in wrapping]
2016-12-03 13:33:31 -08:00
bulat22101 a6f91064a2 pep8: Fix E129 violations 2016-12-03 10:56:36 -08:00
Alex Huang c8ddea16c3 pep8: Fix E122. 2016-12-01 23:16:35 -08:00
AZtheAsian 7e14fe65ab pep8: fix E202 violations 2016-12-01 23:06:02 -08:00
Rafid Aslam c5316b4002 lint: Fix E127 pep8 violations.
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
Tommy Ip e4091c6413 pep8: Fix E222 violations. 2016-11-30 21:49:02 +00:00
Rishi Gupta 4b183cd526 domain migration: Remove several instances of get_realm.
Remove the easy to remove instances of get_realm.
2016-11-26 15:19:56 -08:00
Anders Kaseorg 207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Steve Howell 78474c8bee Add test users with non-lowercase emails.
We are prone to case-sensitivity bugs, so I added AARON and ZOE.
Also, for good measure, I insert them in non-alphabetical order
to try to drive out bugs from non-consistent sorting of user ids.
2016-11-26 11:48:52 -08:00
Tim Abbott 61816f6151 populate_db: Remove replay_old_messages code.
Now that we have the data import/export tool, that's a better
mechanism for importing data than this old and likely buggy code was.
2016-11-15 22:34:44 -08:00
paxapy b90057d557 render_messages.py: render first version of edited messages.
This decreases the number of spurious differences in rendered content
significantly.
2016-11-15 13:39:11 -08:00
Rishi Gupta 5681ee90fa models.py: Rename split_email_to_domain to email_to_domain.
In order to better match models.email_to_username.
2016-11-11 15:26:51 -08:00
Rishi Gupta c1713c9659 Prevent code from using email domain to determine realm when subdomains.
Also removes the intermediate step of going through Realm.domain in the
non-subdomains case. Part of a larger project to remove Realm.domain
entirely.
2016-11-11 15:26:51 -08:00
Umair Khan e127325fce Django 1.10: Update zilencer urls. 2016-11-10 16:20:03 +05:00
paxapy 93965a8e89 Add tools for dumping and comparing markdown renderings.
This adds a couple new tools that can be used to determine whether a
particular change in Zulip's backend markdown processor would impact
the rendering of historical messages, without a human actually looking
at the message content.  This is a useful way to verify whether a
change to our markdown syntax is likely to create problems.

[commit message and code tweaked by tabbott]
2016-11-07 07:49:56 -08:00
Umair Khan 682aa1f298 Django 1.10: Use add_argument for options in BaseCommand. 2016-11-04 10:20:23 -07:00
Umair Khan b140236fcf Django 1.10: Do not use patterns function. 2016-11-04 10:06:00 -07:00
Rishi Gupta c0f1b5d9f6 Change Realm defaults to Community defaults.
Previously, we set restrict_to_domain and invite_required differently
depending on whether we were setting up a community or a corporate
realm. Setting restrict_to_domain requires validation on the domain of the
user's email, which is messy in the web realm creation flow, since we
validate the user's email before knowing whether the user intends to set up
a corporate or community realm. The simplest solution is to have the realm
creation flow impose as few restrictions as possible (community defaults),
and then worry about restrict_to_domain etc. after the user is already in.

We set the test suite to explictly use the old defaults, since several of
the tests depend on the old defaults.

This commit adds a database migration.
2016-11-03 13:53:01 -07:00
Steve Howell af6512e782 Add feedback bot to the test database. 2016-11-03 09:42:27 -07:00
Rishi Gupta db0e509422 do_create_realm: Replace domain argument with string_id.
Turns string_id into a required argument, and domain into an optional
argument.
2016-11-02 22:46:34 -07:00
Rishi Gupta 64bcd71d6e models.Realm: Rename subdomain to string_id.
Does a database migration to rename Realm.subdomain to
Realm.string_id, and makes Realm.subdomain a property.  Eventually,
Realm.string_id will replace Realm.domain as the handle by which we
retrieve Realm objects.
2016-11-02 22:46:34 -07:00
Tomasz Kolek 4790316b57 Add user setting option to always send push notifications.
Add option in user's settings for getting mobile push notifications
even if a Zulip browser is online.  Default is False.

Fixes: #1596.
2016-10-25 10:52:29 -07:00
Rishi Gupta 537ee75761 Create RealmAlias entries for Realm.domain.
This is a preliminary step towards eliminating the realm.domain field
in favor of realm.subdomain.  Includes a database migration to create
these for existing realms.
2016-10-25 10:06:39 -07:00
Steve Howell d21d443d29 Deprecate render_old_messages command. 2016-10-04 11:31:21 -07:00
Steve Howell 7fb992dba3 Simplify and "fix" render_old_messages management command.
The command to render old messages now looks for all messages
not matching the bugdown version, and it no longer directly calls
into model code.  We should still be extremely cautious about
using this code.
2016-10-04 11:31:20 -07:00
hackerkid ea39fb2556 Add option for hosting each realm on its own subdomain.
This adds support for running a Zulip production server with each
realm on its own unique subdomain, e.g. https://realm_name.example.com.

This patch includes a ton of important features:
* Configuring the Zulip sesion middleware to issue cookier correctly
  for the subdomains case.
* Throwing an error if the user tries to visit an invalid subdomain.
* Runs a portion of the Casper tests with REALMS_HAVE_SUBDOMAINS
  enabled to test the subdomain signup process.
* Updating our integrations documentation to refer to the current subdomain.
* Enforces that users can only login to the subdomain of their realm
  (but does not restrict the API; that will be tightened in a future commit).

Note that toggling settings.REALMS_HAVE_SUBDOMAINS on a live server is
not supported without manual intervention (the main problem will be
adding "subdomain" values for all the existing realms).

[substantially modified by tabbott as part of merging]
2016-09-27 23:24:14 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
Tomasz Kolek 0e192d6aff Add colors to streams in development server fixtures.
Fixes: #1454.
2016-08-26 22:38:14 -07:00
Taranjeet Singh 230cc228c2 profile_request.py: Fix line with length greater than 120.
This updates file zilencer/management/commands/profile_request.py.
2016-08-19 11:56:43 -07:00
Rishi Gupta 708b416ca1 Terms of Service: Fix corner cases around new users being created.
This fixes a few bugs in 7910a6e134,
related to automatically created user accounts.
2016-08-13 00:19:54 -07:00
Tim Abbott 95aee366c8 Enable local browser error reporting if ZILENCER_ENABLED. 2016-08-11 13:49:18 -07:00
Tim Abbott ba7ef1c82d zilencer: Fix regression in URLs.
When we refactored zilencer to use a single urls.py file in
bf50dd7771, we accidentally lost the
prefix on the API urls.

This broke sending error report emails if zilencer was enabled.
2016-08-11 12:38:42 -07:00
Tim Abbott 0ca7af6137 Annotate zilencer tests. 2016-08-04 15:53:23 -07:00
Tim Abbott 1f2b3588d2 Annotate zilencer/models.py and fix str types in zilencer views. 2016-08-04 15:53:23 -07:00
Taranjeet Singh 0cafd9268d zilencer/management/commands/populate_db.py: Fix line with len > 120. 2016-07-18 14:42:39 -07:00
Eklavya Sharma 90f306421f Replace bytes by strings in zilencer/migrations. 2016-07-04 12:08:37 -07:00
Eklavya Sharma 88354ad1fc zilencer's populate_db.py: Remove unnecessary cast to text_type. 2016-06-29 10:36:50 -07:00
Eklavya Sharma e155573a1c zilencer's populate_db.py: Type annotate all variables. 2016-06-27 10:22:49 -07:00
Eklavya Sharma 8cacd5fcf8 zilencer's populate_db.py: Type annotate some variables. 2016-06-27 21:00:10 +05:30
Eklavya Sharma ea3c671494 zilencer's populate_db.py: Improve an annotation. 2016-06-27 21:00:10 +05:30
Eklavya Sharma 66d1867869 zilencer's populate_db.py: Fix string types. 2016-06-27 21:00:10 +05:30
acrefoot e4ed9195dc Remove rest_dispatch hack and optimize imports.
For a long time, rest_dispatch has had this hack where we have to
create a copy of it in each views file using it, in order to directly
access the globals list in that file.  This removes that hack, instead
making rest_dispatch just use Django's import_string to access the
target method to use.

[tweaked and reorganized from acrefoot's original branch in various
ways by tabbott]
2016-06-24 16:11:03 -07:00
Tomasz Kolek f8c368c07f Add default-bot during populate_db. 2016-06-21 11:47:38 -07:00
Eklavya Sharma d169cc5376 zerver/lib/bulk_create.py: Fix string annotations.
Change string type from `str` to `text_type` where required.
Also fix clashing annotations.
2016-06-13 10:08:14 +05:30
Tomasz Kolek 2ac9c792f3 test_hooks: Use an incoming webhook bot for webhook tests. 2016-06-07 21:27:04 -07:00
Evan Palmer bd0fd61821 Annotate zilencer.error_notify.
Also fix the annotation of zilencer.views.report_error.

The `report` arguments are a Dict containing both strings and the
`more_info` sub-dictionary, so we type them as Dict[str, Any].

[tweaked by tabbott]
2016-06-07 13:44:56 -07:00
pxhanus e247d9783f Annotate zilencer/views.py. 2016-06-04 22:07:01 -07:00
Tim Abbott f2320bf27f Fix populate_db send_messages annotation. 2016-06-04 16:11:14 -07:00
medullaskyline 7c2c7fb31c Annotate zerver/lib/bulk_create.py. 2016-06-04 15:51:05 -07:00
Tim Abbott a1a27b1789 Annotate most Zulip management commands. 2016-06-04 10:12:06 -07:00
Ashish Kumar 31bf6b8259 Type annotation of zerver/models.py
[Substantially revised by tabbott]

This probably still has some bugs in it, but having mostly complete
annotations for models.py will help a lot for the annotations folks
are adding to other files.
2016-06-02 23:28:34 -07:00
Umair Khan 08fbd57245 [i18n] Make error messages translatable.
Make all strings passing through `json_error` and `JsonableError`
translatable.

Fixes #727
2016-05-31 07:40:42 -07:00
Vishnu Ks 100d885f23 Change default announcement stream to announce.
Fixes #788.
2016-05-31 07:38:07 -07:00
Tim Abbott 960144a49e Desupport using uninstantiated REQ with has_request_variables.
This makes life difficult for doing static type annotations, and
didn't make the code look that much better anyway.
2016-05-31 07:31:15 -07:00
Eklavya Sharma 94e4b39112 Replace python2.7 by python everywhere. 2016-05-29 05:03:08 -07:00
Tomasz Kolek 8c18b8947f Add bot_type field to UserProfile.
This is intended to support creating different types of bots with
potentially limited permissions.
2016-05-19 22:37:37 -07:00
Umair Khan f9bbc5d6ff Enable i18n support in URL configuration.
This supports i18n using all of the following:
- I18N urls
- Session
- Cookie
- HTTP header
2016-05-19 08:33:30 -07:00
Tim Abbott bf50dd7771 Simplify zilencer urls.py configuration to a single file.
The previous separated-out configuration wasn't helping us, and this
makes it easier to make the extra installed applications pluggable in
the following commits.
2016-04-26 21:35:07 -07:00
Tim Abbott 49799440a4 Replace use of django-guardian with fields on UserProfile.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.

As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.

When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.

We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.

Fixes #441.
2016-04-20 21:51:52 -07:00