Long ago, there was work on an experimental integration model where
every user in a realm would have administrative control over all bots,
with the goal of simplifying the process of setting up communally
administered bots for smaller teams. While that new model was never
fully implemented (and thus never setup as an option), an error in
that original implementation meant that the data on all bots in a
realm, including their API keys, was sent to the browsers of users via
the `realm_bots` variable in `page_params`. The data wasn't displayed
in the UI for non-admin users, but was available via e.g. the
javascript console.
This commit updates this behavior to only send sensitive bot data like
API keys to the owner of the bot (and realm admins).
We may in the future implement a model simplifying communally
administered integrations, but if we do that, those bots should be
limited in their capabilities (e.g. only able to send webhook
messages).
This bug has been present since Zulip was released as open source.
Previously we relied on having two matching list of fields for the
get_active_user_dicts_in_realm, one in the actual code and the other
in the caching system. By unifying these lists to have a single
source, we eliminate a class of caching bugs we might otherwise
regularly introduce.
This commit adds the capability to keep track and remove uploaded
files. Unclaimed attachments are files that have been uploaded to the
server but are not referred in any messages. A management command to
remove old unclaimed files after a week is also included.
Tests for getting the file referred in messages are also included.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.
As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.
When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.
We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.
Fixes#441.
Add a function email_allowed_for_realm that checks whether a user with
given email is allowed to join a given realm (either because the email
has the right domain, or because the realm is open), and use it
whenever deciding whether to allow adding a user to a realm.
This commit is not intended to change any behavior, except in one case
where the Zulip realm's domain was not being converted to lowercase.
If the content wasn't rendered, both rendered_content and
rendered_content_version would be None. In addition to being
confusing, in Python 3, `None < 2` is an error and this code breaks.
New users will no longer get desktop and audible notifications for all streams
by default.
This also updates the `day1` follow-up email to let users know they can
customize how and when Zulip notifies them of new messages.
Lastly, this adds a `changelog.md` file, following the conventions from
keepachangelog.com, to track changes for new releases.
This commit loses some indexes, unique constraints etc. that were
manually added by the old migrations. I plan to add them to a new
migration in a subsequent commit.
(imported from commit 4bcbf06080a7ad94788ac368385eac34b54623ce)
Include new field on Realm to control whether e-mail invitations are required
separately from whether the e-mail domain must match.
Allow control of these fields from admin panel.
Update logic in registration page to use these fields.
(imported from commit edc7f0a4c43b57361d9349e258ad4f217b426f88)
Also increase the number of messages sent as context from 5 to 10 and
look up to 15 minutes in to the past for context.
(imported from commit bfaed9bcff1ee2047fc3b7a63acf93cd2d47cc7d)
Resolves a AmbiguousTimeError.
Approved by Leo.
This reverts commit ebfaeb97ffda22b618be7a9206877f9d2ec53404.
(imported from commit 42b29c6c57eb954952a740bc89611031cef1834a)
This can be used by mirroring scripts to only forward messages to users
who do not zulip accounts.
(imported from commit 200d6bcaaf39238bfb01480a9e906d567d4d9e11)
When you are at mentioned in a stream we will now send you up to the
last five messages which were sent in the past 5 minutes on the same
topic and stream.
(imported from commit 6df6c1cf868722a7bf76e54710e38741a7ac8f31)
Allow bot owners to set which streams their will receive events for
without needing to change a configuration file.
(imported from commit 2b69e519dbc12ffbdba072031a7f7196c9e50e33)
This allows bot owners to configure which streams messages are delivered
to without needing to change webhook URLs or configuration files.
(imported from commit 32a0c26657c145b001cd8cb3ce0a0364d48902ce)
Before saving a Message object, call update_calculated_fields()
to set the has_attachment/has_image/has_link fields.
Note that the pre_save hook we added here does not get called
if you call bulk_create, hence the explicit call to
update_calculated_fields() in do_send_messages().
(imported from commit 1d60ae5908ef186aa5ff1e39277dbb2b765e60d4)
A stream is vacant when it has no subscribers and occupied when it has at least
one subscriber.
We have a slightly odd model where stream creation is conflated with
subscription creation. Streams are created by attempting to subscribe to a
stream that doesn't exist. We also hide streams with no subscribers from users
to make it seem like they've gone away. However, we can't actually remove those
streams because we want to preserve history.
This commit moves us towards a separation of these two concepts. By sending
events for stream creation, occupation, vacancy, and deletion, we allow clients
to directly observe the global state of streams rather than indirectly observing
subscription information. A more complete solution would involve adding a view
for explicitly creating streams without subscribing to them.
This commit does not handle the intricacies of invite-only streams. We
currently simply do not send these events for invite-only streams.
(imported from commit 5430e5a5eecefafcdba4f5d4f9aa665556fcc559)
We will need to run these commands manually when deploying to staging:
CREATE INDEX CONCURRENTLY "zerver_message_has_attachment" ON "zerver_message" ("has_attachment");
CREATE INDEX CONCURRENTLY "zerver_message_has_image" ON "zerver_message" ("has_image");
CREATE INDEX CONCURRENTLY "zerver_message_has_link" ON "zerver_message" ("has_link");
(imported from commit 84808dc6b1af887ddf784cb8a875ae462f4df985)
This commit makes it so that we don't individually query
auth_permission for every user on the realm during calls
to Realm.get_admin_users(). This should speed up page loads.
To apply this commit, we had to upgrade all of our servers
with this patch to django-guardian:
https://github.com/lukaszb/django-guardian/pull/178#issuecomment-31049062
(imported from commit a1604bf573a5005c9abc128a680a7da6a20cabef)
This allows clients to show an indication if users are on mobile or can
be reached via mobile.
(imported from commit 819c27a141f71dee32a97fd5c1ac405b039a5038)
Also rename the function, which was previously called to_presense_dict.
This will let us be more flexible about adding additional values.
(imported from commit 323b7d2df53918928190c9ee9544b4599a5e2df3)
From a user's perpsective, the stream has been deleted. From the
database's perspective, the stream has been deactivated -- the stream
messages still exist.
(imported from commit b08b30b2a822663e17d64182af1fb160c2193344)
This is a slight behavior change, as we now flush user_profile
caches for bots as well as humans.
(imported from commit 24c72c44d851ee4c66a67a4728cd6c548faeedcd)
A description was added to the streams and it is now displayed on the
subscriptions page. It can not be set in the UI yet.
(imported from commit 81d08b65eee42dba87cd99dd5bd30106c4eb6c6a)
Added a default_desktop_notifications boolean to userprofile with a UI
in Zulip Labs. This flag is used to default the notification flag on new
subscriptions.
(imported from commit a25223cc5ecf09980cf877991e25034bb3fd4046)
Note that this doesn't actually restrict anybody yet, but it
makes it so that UserProfile.can_create_streams must return True
for a user to create a stream. We can modify that in the future
to have special behavior for realms that want more restrictions.
(imported from commit 432e85b1ca86aaee4a6bd1d4a6d0b2c78ecb0863)
This is for the CUSTOMER28 folks, so that they can turn Zulip into a more "chat client" thing.
(imported from commit 373a8afae4998fce5560e7b2bd13804c8fbb39fc)
This replaces the AppleDeviceToken table with a generic
PushDeviceToken with a `kind` field to make it easier to add functionality
like per-device/per-stream settings that share code between Android and
iOS devices.
The schema must continue to work on prod with the old table name, so we
add the new table in parallel and can drop the old table once this code
hits prod and any necessary data is copied.
(imported from commit 0209a7013f2850ac6311f23c3d6f92c65ffd19e3)
This allows us to avoid a circular import when importing models.py
from inside bugdown for the realm-filters-from-database branch.
(imported from commit 7de85b54243132ade6818b080abdc8c5e8ad84f5)
Now that we support email aliases, we have to be careful when going from
an email address to a domain that we assume we can use to get a Realm
object for. When we care about the Realm's domain, we want to follow
any RealmAliases that exist for a certain domain.
When we just care about the original email address domain itself,
for comparison or other purposes, use split_email_from_domain
This removes the ambiguity of having to decide when to use
email_to_domain + RealmAlias or just email_to_domain
(imported from commit 0e199495502d946ce2e1aae56263e7e8665be4ed)
Client objects are immutable and there are very few of them, so caching them in
memory is cheap and saves a trip to memcached.
(imported from commit 300b9b402f4e509f86a7fd86b5f898dc3f43738f)
UserProfile.show_admin was intended to be a check for users that have
administrative rights in other realms, which we've harmlessly but
erroneously been using to check if they are an admin in their realm.
Use the more straightforward check instead, with a more intuitive
name.
(imported from commit d81050c7dbbb19e59c5e31750be303a4630e1456)
ScheduledJobs with type Email displace the usual mandrill codepaths
in the Zulip Enterprise deploys
* Email-specific helper functions will appear in deliver_email.py
* 0058_auto__add_scheduledjob.py
(imported from commit 8db08d8a279600322acfdbed792dc1a676f7a0ab)
If a user types "/me runs to the store", we put "runs to the store"
in bold after their name.
(imported from commit fbc11e99244e1c8fa1c03e4753e706957fcd449e)
I'm pretty sure this code was a hack to workaround the fact that
without database-level autocommit, .get_or_create is racy.
(imported from commit bfdd476a354aee439feeaa0030ae59cc43c165b9)
This was a precursor to UserMessage.flags.read that never got used
because we decided to use django-bitfield.
(imported from commit 868754723c07ee9b85ae951aee785e571ccfef97)
Here we introduce a new Django app, zilencer. The intent is to not have
this app enabled on LOCALSERVER instances, and for it to grow to include
all the functionality we want to have in our central server that isn't
relevant for local deployments.
Currently we have to modify functions in zerver/* to match; in the
future, it would be cool to have the relevant shared code broken out
into a separate library.
This commit inclues both the migration to create the models as well as a
data migration that (for non-LOCALSERVER) creates a single default
Deployment for zulip.com.
To apply this migration to your system, run:
./manage.py migrate zilencer
(imported from commit 86d5497ac120e03fa7f298a9cc08b192d5939b43)
This has a small bug where we don't actually filter the message out of
the home view; fixing that requires adding an index on the "flags"
field of UserMessage.
(imported from commit 492c99d0a8e87b253e577be6564bec12099bd8e9)
In order to support iOS Push Notifications, we need to keep track
of a device's unique APNS Token. These are delivered to our iOS
code after registering for remote notifications
(imported from commit bbe34483e1380dc20a1c93e3ffa1fcfdb9087e67)
When we don't already have old messages in cache, we need to
fetch data from the database and create dictionaries for the
cache. This commit makes that process work in 50ms, instead
of 130ms, for the data set in test_bulk_message_fetching(),
which is 602 records. Before this commit we had about 132
microseconds of unnecessary churn per message, because we
were fetching DB fields we didn't need and incurring the cost
of the Django ORM. Now we use values() to get only the columns
we need, and we take advantage of previous commits that make
our code less OO and more function-driven, so we can pass the
values directly to build_message_dict() without having to create
objects.
A couple caveats on this commit:
1) I haven't been able to get good measurements on the overall
effect on get_old_messages_backend(). If you kill the cache to
force DB queries, you introduce noise related to sessions and
user profiles.
2) Look at the long comment in this commit related to
re-rendering messages in this codepath. The problem precedes
this commit.
(imported from commit dcb64aa9416f0e9583355ddd6dc3adfa746b9fc7)
Only call a function on the message object in the unfortunate
situation that we are rendering new content in to_dict_uncached().
Long term, it would be nice if this function didn't have side
effects, and we had a better strategy for upgrading rendered
content when bugdown versions change.
(imported from commit 2a323f52af37a6d651c171cb8234fbfa3d25d561)
This function doesn't require the whole UserProfile object to
create the avatar url, and we call it from Message.to_dict_uncached().
(imported from commit e814caab101c4fedd1ba66df041a3408014e4085)
For a bunch of self-dot references, move them to the top. (This
is kind of funny out of context, but it sets us up for future
refactorings.)
(imported from commit 4ebc1c44a633d86772df1828c51180707769c3dc)
If this line of code were ever called, it would crash anyway,
because it would be an unknown type, and Recipient.type_name()
would raise a KeyError.
(imported from commit db38c5f71fb2f0b044a832eb88e53fceb0d8a9cf)
This is a variation of get_display_recipient that takes
values instead of an object, so that it is decoupled from
the Django object system.
(imported from commit 25bed43ecd62f1fe0176d517b7003e7f4c78bc37)
If it's ok for the tests to use memcached, it should be ok
for them to use the in-process cache too.
(imported from commit be43879c3c48f3780317fd5b4139b44d4a1f0ed3)
This is a harmless extraction designed to allow subclasses override
the behavior of how rendered content gets saved.
(imported from commit 9df4ed9f86c857897fcb5f2b6781bfc5a0813766)
The get_status_dict_by_realm helper gets called whenever our
realm user_presences cache expires, and it used to query these fields:
"zerver_userpresence"."id", "zerver_userpresence"."user_profile_id", "zerver_userpresence"."client_id", "zerver_userpresence"."timestamp", "zerver_userpresence"."status", "zerver_userprofile"."id", "zerver_userprofile"."password", "zerver_userprofile"."last_login", "zerver_userprofile"."is_superuser", "zerver_userprofile"."email", "zerver_userprofile"."is_staff", "zerver_userprofile"."is_active", "zerver_userprofile"."is_bot", "zerver_userprofile"."date_joined", "zerver_userprofile"."bot_owner_id", "zerver_userprofile"."full_name", "zerver_userprofile"."short_name", "zerver_userprofile"."pointer", "zerver_userprofile"."last_pointer_updater", "zerver_userprofile"."realm_id", "zerver_userprofile"."api_key", "zerver_userprofile"."enable_desktop_notifications", "zerver_userprofile"."enable_sounds", "zerver_userprofile"."enter_sends", "zerver_userprofile"."enable_offline_email_notifications", "zerver_userprofile"."last_reminder", "zerver_userprofile"."rate_limits", "zerver_userprofile"."avatar_source", "zerver_userprofile"."tutorial_status", "zerver_userprofile"."onboarding_steps", "zerver_userprofile"."invites_granted", "zerver_userprofile"."invites_used", "zerver_userprofile"."alert_words", "zerver_userprofile"."muted_topics", "zerver_client"."id", "zerver_client"."name"
Now it queries just the fields it needs:
"zerver_client"."name", "zerver_userpresence"."status", "zerver_userpresence"."timestamp", "zerver_userprofile"."email" FROM "zerver_userpresence"
Also, get_status_dict_by_realm is now namespaced under UserPresence as a static method.
(imported from commit be1266844b6bd28b6c615594796713c026a850a1)
These engagement data will be useful both for making pretty graphs of
how addicted our users are as well as for allowing us to check whether
a new deployment is actually using the product or not.
This measures "number of minutes during which each user had checked
the app within the previous 15 minutes". It should correctly not
count server-initiated reloads.
It's possible that we should use something less aggressive than
mousemove; I'm a little torn on that because you really can check the
app for new messages without doing anything active.
This is somewhat tested but there are a few outstanding issues:
* Mobile apps don't report these data. It should be as easy as having
them send in update_active_status queries with new_user_input=true.
* The semantics of this should be better documented (e.g. the
management script should print out the spec above)x.
(imported from commit ec8b2dc96b180e1951df00490707ae916887178e)
This cache was created to make recipient lookups within a single
request (e.g. when fetching old messages) cheaper. To support stream
name changes, we need to invalidate this cache on every request so
that users get a consistent view of the name change.
(imported from commit 801051b9f6a108c1f50be7eca9a1242d661919b1)
This is experimental, for staging only. There might be a better
way to model this than dueling force_expand/force_collapse flags,
but it works for now. The code in collapse_recipient_group()
could also be DRYed up relative to expand_summary_row().
(imported from commit 107151d1ecd640970fb7700d41278a003bd1abaa)
This change will allow us to test the muting feature on
staging. Any topic named "muted" will automatically be
muted. You can also mute any other topic on the console:
muting.mute_topic('devel', 'ios');
current_msg_list.rerender();
More UI around this experiment will be coming soon, as well
as support for muting entire streams.
The muting module keeps track of which topics are muted, but a
user can expand muted messages, and once that happens, the
messages are marked with the "force_expand" flag that gets
persisted to the back end.
Muted messages are rendered in similar fashion to the summarized
rows, and as part of unifying some of that code, we have
made it so that expanding a summarized section doesn't remove
individual flags related to summaries; instead, the messages
get the force_expand flag set.
(imported from commit acee4190e63813d46850415c41ff8ebfae4a6953)
This commit CANNOT be deployed until the previous schema change
([schema] models: add an email_token field to Streams) is on prod.
Before applying this schema change, run the populate-stream-tokens
management command to generate tokens for streams that need them.
(imported from commit 7adc81c8c317ec5d59dd59ba42a4dc1a46174007)
We now show a list of users and allow you to deactivate a user using the
same process as `python manage.py deactivate_user`.
We add a new menu item accessible from the gear icon which will eventually
have much more than just this, but we have a good start here.
Here we also add a property to UserProfile which determines whether you're
eligible to access the administration panel, and then have code which shows
the menu option if so.
This introduces a new JS file, admin.js.
(imported from commit 52296fdedb46b4f32d541df43022ffccfb277297)
The e-mail forwarder will use this. Set it to nullable temporarily to
accomodate existing streams; later commits will a) provide a script to
give all streams a token, and b) make the field non-null.
Realm administrators will eventually have a UI to regenerate stream
tokens.
(imported from commit a084d0a7012eb9665e4da095cbc46aa9ef354eaa)
It will have new callers soon and that's as fine a place as any to
avoid circular imports.
(imported from commit 089a724e9ad06cb5a51ffe80f1729d789238e5f6)
When you read messages in a narrow and then un-narrow, collapse
adjacent messages read in the narrow into a summary row that can
be clicked to expand those messages.
Scoped to staging with feature flags.
The implementation of this within our current MessageList is rather ugly.
(imported from commit bcb3a39d8c0c334136fe86318f18ead03f0f50bf)
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).
At the time it is deployed, we need to make a few changes directly in
the database:
(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';
(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
Vishnu Ks
Tomasz Kolek
Tim Abbott
Aristeidis Fkiaras
rahuldeve
Vladislav Manchev
Varshit
Ryan Moore
Reid Barton
Nicholas Bergson-Shilcock
Tim Abbott
David Roe
Kate Buckner
Jason Michalski
Leo Franchi
Luke Faraone
Steve Howell
Zev Benjamin
Jessica McKellar
Leo Franchi
acrefoot
Kevin Mehall
Waseem Daher
Tim Abbott
Jessica McKellar
acrefoot
Scott Feeney
Kevin Mehall