This commit implements support for copying over static files
for all bots in the zulip_bots package to
static/generated/bots/ during provisioning. This directory
isn't tracked by Git. This allows us to have access to files
stored in an arbitrary zulip_bots package directory somewhere
on the system. For now, logo.* and doc.md files are copied over.
This commit should act as a starting point for extending our
macro-based Markdown framework to our bots/API packages'
documentation and eventually rendering these static files
alongside our webhooks' documentation.
The motivation for this is that we'll want to use the STATIC_ROOT
variable in this code in the upcoming commits.
While we're at it, we give it a proper section in the file.
This enforces our use of a consistent style in how we access Python
modules; "from os.path import dirname" is a particularly popular
abbreviation inconsistent with our style, and so it deserves a lint
rule.
Commit message and error text tweaked by tabbott.
Fixes#6543.
This commit enables user to authenticate with any attribute set in
AUTH_LDAP_USER_SEARCH given that LDAP_EMAIL_ATTR is set to an email
attributes in the ldap server. Thus email and username can be
completely unrelated.
With some tweaks by tabbott to squash in the documentation and make it
work on older servers.
This is necessary for tests the to pass with the new config_error code
path, which we don't want to trigger except on specific tests for
those features.
Since we're auto-detecting the value anyway, there's no reason it
can't be moved to DEFAULT_SETTINGS.
This lets us remove some clutter from the installation documentation.
This adds the authors to the Zulip repository on GitHub from
/authors/ along with re-styling the page to fit the same
aesthetic as /for/open-source/ and other product-pages.
The new endpoints are:
/json/mark_stream_as_read: takes stream name
/json/mark_topic_as_read: takes stream name, topic name
The /json/flags endpoint no longer allows streams or topics
to be passed in as parameters.
With this change, we get as far as printing the message
"APNS: Sending apple push notification to devices" to the
log when a recent TestFlight build of the app is due for
a notification, and then don't hit an exception. But
on the other hand I still don't get an actual notification
on my phone, so there's still some debugging to do.
Here are the functions in top_left_corner:
get_global_filter_li: pure code move
update_count_in_dom: simplifed copy of similar function in stream_list.js
update_dom_with_unread_counts: pure code move, split out from function
of same name in stream_list.js
delselect_top_left_corner_items: pure code move
handle_narrow_activated: pure code move + rename
handle_narrow_deactivated: pure code move, split out from from function
of smae name in stream_list.js
I was too hasty in pushing this -- it looks right logically, but it
breaks a test. May not be hard to fix forward, but reverting now to
unbreak the build in master.
This reverts commit 02acd467b4.
This change simplifies how we mark all messages as read. It also
speeds up the backend by taking advantage of our partial index
for unread messages. We also use a new statsd indicator.
This commit extract send_messages.js to clean up code related
to the following things:
* sending data to /json/report_send_time
* restarting the event loop if events don't arrive on time
The code related to /json/report changes the following ways:
* We track the state almost completely in the new
send_messages.js module, with other modules just
making one-line calls.
* We no longer send "displayed" times to the servers, since
we were kind of lying about them anyway.
* We now explicitly track the state of each single sent
message in its own object.
* We now look up data related to the messages by local_id,
instead of message_id. The problem with message_id was
that is was mutable. Now we use local_id, and we extend
the local_id concept to messages that don't get rendered
client side. We no longer need to react to the
'message_id_changed' event to change our hash key.
* The code used to live in many places:
* various big chunks were scattered among compose.js,
and those were all moved or reduced to one-line
calls into the new module
* echo.js continues to make basically one-line calls,
but it no longer calls compose.report_as_received(),
nor does it set the "start" time.
* message_util.js used to report received events, but
only when they finally got drawn in the home view;
this code is gone now
The code related to restarting the event loop if events don't arrive
changes as follows:
* The timer now gets set up from within
send_messages.message_state.report_server_ack,
where we can easily inspect the current state of the
possibly-still-in-flight message.
* The code to confirm that an event was received happens now
in server_events.js, rather than later, so that we don't
falsely blame the event loop for a downstream bug. (Plus
it's easier to just do it one place.)
This change removes a fair amount of code from our node tests. Some
of the removal is good stuff related to us completing killing off
unnecessary code. Other removals are more expediency-driven, and
we should make another sweep at ramping up our coverage on compose.js,
with possibly a little more mocking of the new `send_messages` code
layer, since it's now abstracted better.
There is also some minor cleanup to echo.resend_message() in this
commit.
See #5968 for a detailed breakdown of the changes.
This file hasn't reflected the actual configuration of any live
installation for some time, nor been part of any tests or other
mechanism to regularly validate it, so it's naturally fallen
behind as we make changes to the set of settings and typically
don't update this file accordingly. Just remove it; all the
documentation functions it serves are already served just as
well by prod_settings_template.py and its ample comments.
Apparently, the filters implementation was doing some sort of strange
caching, where you would need to restart the server in order to
refresh for changes to the markdown content.
We fix this by switching to just calling the render_markdown_path
function from Jinja2.
Fixes#5974.
This completes the major endpoint migrations to eliminate legacy API
endpoints from Zulip.
There's a few other things that will happen naturally, so I believe
this fixes#611.
This new module tracks the recent topic names for any given
stream.
The code was pulled over almost verbatim from stream_data.js,
with minor renames to the function names.
We introduced a minor one-line function called stream_has_topics.
The whole thing is an error, so "message" is a more apt word for the
error message specifically. We abbreviate that as `msg` in the actual
HTTP responses and in the signatures of `json_error` and friends, so
do the same here.
static/ serves static files which get copied around per deploy. Since
the webpack stats files need a consistent name and change per deploy,
they can't live in static/.
This fixes a bug that preventing downgrading a Zulip server to an old
version.
I pushed a bunch of commits that attempted to introduce
the concept of `client_message_id` into our server, as
part of cleaning up our codepaths related to messages you
sent (both for the locally echoed case and for the host
case).
When we deployed this, we had some strange failures involving
double-echoed messages and issues advancing the pointer that appeared
related to #5779. We didn't get to the bottom of exactly why the PR
caused havoc, but I decided there was a cleaner approach, anyway.
This is mostly straightforward moving of code out of compose.js.
The code that was moved currently supports sending time
reports for sent messages, but we intend to grow out the new
module to track more state about sent messages.
The following function names in this commit are new, but their
code was basically pulled over verbatim:
process_success (was process_send_time)
set_timer_for_restarting_event_loop
clear
initialize
All the code in the new module is covered by previous tests that
had been written for compose.js. This commit only modifies
a few things to keep those tests.
The new module has 100% node coverage, so we updated `enforce_fully_covered`.
Prior to this commit, 7 megabytes of images (through 253 individual requests)
were heavily slowing down the initial load. With this commit, we load only the
logos (60 or so images).
Documentation and images for the individual integration sub-pages is requested
separately using the /integrations/doc/ endpoint, which returns HTML.
This redesigns the /help/ page sets to be a single page app that uses
history.pushState to work the same as the old app.
The big new feature is that now we have the index in a nicely designed
left sidebar.
Django 1.11 adds the ability to pass context processors in Jinja2
backend. Django also sends template_rendered signal in tests.
These two issues were the reason why we added Jinja2 backend, but
after upgrading to Django 1.11 we can remove it.
We still need jinja2/__init__.py, which modifies the environment,
and jinja2/compressors.py, which adds minify_js compressor.
Django started supporting context processors from version 1.11; as
a result of this, we can get rid of some of the code which is now
being taken care of by Django.
This system hasn't been in active use for several years, and had some
problems with it's design. So it makes sense to just remove it to declutter
the codebase.
Fixes#5655.
This commit removes the ability to configure different validity durations
for different types of confirmation links. I don't think the extra
configurability was worth the extra complexity, either for the user trying
to understand the settings, or for the developer trying to understand the
code.
The commit replaces all confirmation validity duration settings with a
single setting, settings.EMAIL_CONFIRMATION_DAYS.
The only setting it removes is settings.EMAIL_CHANGE_CONFIRMATION_DAYS,
which was introduced in 5bf83f9 and never advertised in prod_settings.py.
This will allow for customized senders for emails, e.g. 'Zulip Digest' for
digest emails and 'Zulip Missed Messages' for missed message emails.
Also:
* Converts the sender name to always be "Zulip", if the from_email used to
be settings.NOREPLY_EMAIL_ADDRESS or settings.ZULIP_ADMINISTRATOR.
* Changes the default value of settings.NOREPLY_EMAIL_ADDRESS in the
prod_setting_template to no longer have a display name. The only use of
that display name was in the email pathway.
This old third party library added support
for a "mousewheel" event to detect scrolling.
However, it is not compatible with jQuery 3
and is obsolete now that there is a standard
"wheel" event that accomplishes the same thing.
Guardian adds functionality on top of Django auth system to set
per object permissions. Its problem is that it is has poor performance.
So we decided to remove it in release 1.4.0, but we still kept the
option to revert back to an older version which used Guardian.
See commit 49799440a4 for more details.
This commit is the final piece in the string of commits which move
us towards completely removing guardian from our codebase. The way
we do it as follows:
If you are upgrading from a version <= 1.3.10, you first need to
upgrade to 1.4.x (we recommend 1.4.2). The reason is that we
deprecated Guardian in this version. Once you have upgraded to
1.4.x we can be sure that your Zulip installation doesn't depend
on Guardian and all the data has been successfully migrated away from
Guardian. The second step is to upgrade to latest release which will
not include any reference to Guardian in the codebase. After this
commit migrating directly to the latest release will not work because
in that case Guardian data will not migrate.
The backward incompatible change that this introduces is that
we have squashed all the migrations till version 1.4.0. This was
necessary to remove Guardian because it was needed by the reverse
migration. These migrations were from 0001 to 0028.
Fixes#5420
This isn't very slick, but it should get the main points down,
and it's past time we got something like this up. Definitely
needs in the future another pass at the text, and also some images
(screenshots, etc.) and styling.
This makes our `zproject.jinja2.backend.Template` compatible with
Jinja2. After this change we don't need to override __init__ function
in Template class.
The only reason we now need to create our own Template class is that
we need to send template_rendered signals.
We need our own Jinja2 class because we need to maintain backward
compatibility with Django 1.10 and we need inject `debug` parameter.
- Remove `perfect-scrollbar` from `static/third` and fetch it from npm.
- Upgrade `perfect-scrollbar` to 0.7.1.
- Bump up the `PROVISION_VERSION` to 5.6.
Changed `wheelSpeed` in "static/js/scroll_bar.js" to 0.5, because when it
20, the scrollbar scrolls very fast.
Changed 'wheelSpeed' in "static/js/emoji_picker.js" from 25 to 0.68
(based on tabbott's testing of scrolling through the emoji list).
Part of #1709.
This page describes software the user will get from upstream for
their own devices, independent of what's on the server they're
using. So it should live in a place maintained together with
that other software, rather than be distributed and versioned
with the server.
The use of ZILENCER_ENABLED to tell the difference is rather a hack
but is currently how we do this in the small handful of similar
spots; see #5245.
Fixes#5234.
In pm_conversations.js, added function to make a user a PM partner and
another function to check if a user is a PM partner. A PM partner is
someone with whom the user has been in a PM with.
In recent_senders module, added a data structure to hold timestamps of
users' latest message in a topic. Also added a function to compare 2
users based on above timestamp. Added a function to process messages for
the data structure and a call in add_message_metadata. Also added node
tests for insertion of data into recent_senders.senders.
This makes it possible for Zulip administrators to delete messages.
This is primarily intended for use in deleting early test messages,
but it can solve other problems as well.
Later we'll want to play with the permissions model for this, but for
now, the goal is just to integrate the feature.
Note that it saves the deleted messages for some time using the same
approach as Zulip's message retention policy feature.
Fixes#135.
Previously our scope setting only allowed us access to
publicly listed email addresss. This commit changes that
to get access to private email addresses as well.
Fixes: #4937.
If a Markdown macro contains Jinja2 template code, it isn't rendered
because render_markdown_path calls template.render on the including
.md file before the macro has been included. And then the including
.md file is converted to HTML. Therefore, the template code within
a Markdown macro (if any) never gets rendered and is returned as it is.
Now, after the source .md file is converted to HTML,
render_markdown_path renders the resulting HTML so that any template
code within included macros (if any) is finally rendered.
This is one of the last major endpoints that were still done in the
pre-REST style.
While we're at it, we change the endpoint to expect a stream ID, not a
stream name.
We used to use constructions like
from_email = "Zulip <%s>" % (settings.NOREPLY_EMAIL_ADDRESS,)
but no longer do. All references to settings.NOREPLY_EMAIL_ADDRESS in the
codebase now do not append a display name.
Specifically, this makes easily available to the desktop and mobile
apps data on the server's configuration, including important details
like the realm icon, name, and description.
It deprecates /api/v1/get_auth_backends.
- Add file_name field to `RealmEmoji` model and migration.
- Add emoji upload supporting to Upload backends.
- Add uploaded file processing to emoji views.
- Use emoji source url as based for display url.
- Change emoji form for image uploading.
- Fix back-end tests.
- Fix front-end tests.
- Add tests for emoji uploading.
Fixes#1134
Modified composebox_typeahead.js to recognize the triple backtick
and tilde for code blocks, and added appropriate typeahead functions
in that file and in typeahead_helper.js.
Additionally, a new file pygments_data.js contains a dictionary of
the supported languages, mapping to relative popularity
rankings. These rankings determine the order of sort of the
languages in the typeahead.
This JavaScript file is actually in static/generated/pygments_data.js, as it
is generated by a Python script, tools/build_pymgents_data.py. This is
so that if Pygments adds support for new languages, the JavaScript file
will be updated appropriately. This python script uses a set of popularity
rankings defined in lang.json.
Corresponding unit tests were also added.
Fixes#4111.
This completes a major redesign of the Zulip login and registration
pages, making them look much more slick and modern.
Major features include:
* Display of the realm name, description and icon on the login page
and registration pages in the subdomains case.
* Much slicker looking buttons and input fields.
* A new overall style for the exterior of these portico pages.
Despite the length of this commit, it is a very straightforward
moving of code from narrow.js -> narrow_state.js, and then
everything else is just s/narrow.foo()/narrow_state.foo()/
(with a few tiny cleanups to remove some code duplication
in certain callers).
The only new functions are simple setter/getters that
encapsulate the current_filter variable:
narrow_state.reset_current_filter()
narrow_state.set_current_filter()
narrow_state.get_current_filter()
We removed narrow.predicate() as part of this, since it was dead
code.
Also, we removed the shim for narrow_state.set_compose_defaults(),
and since that was the last shim, we removed shim.js from the app.
This code makes the right pane work in "Manage Streams" when
you are editing a stream subscription. It handles basic
functionality (submitting forms, etc.), live updates, and
showing the pane as needed.
Most of the code here was simply moved from subs.js, but some
functions were pulled out of larger functions:
live update:
add_me_to_member_list
update_stream_name
update_stream_description
collapse/show:
collapse
show_sub
We also now export subs.show_subs_pane.
We eventually want stream_edit not to call into subs.js, and
this should be fairly easy--we just need to move some shared
methods to a new module.
This new modules handles the UI to create streams. It mostly moves
code from subs.js.
It introduces an API around what used to be called meta.stream_created:
reset_created_stream()
set_name()
get_name()
It only partially moves new_stream_clicked().
This fixes a performance problem where we were previously starting up
a full Django process (~0.7s even on a fast machine) every time a new
email came in, potentially allowing users to accidentally DoS a Zulip
server. Now, we just post over HTTPS, allowing the existing thread
pool support to do its job.
- Add script wrapper to communicate postfix pipe with django web server
over HTTP(S). It uses shared_secret authentication mode.
- Add django view to process messages from email mirror server.
- Clean management command `email-mirror`. Left just functional
for cron email processing.
- Add routes for new tornado view.
- Change pipe script in master process postfix config template
based on updated script.
- Add tests.
Tweaked by tabbott to adjust the directory and set better defaults.
Fixes#2421.
This is an incomplete cleaned-up continuation of Lisa Neigut's push
notification bouncer work. It supports registration and
deregistration of individual push tokens with a central push
notification bouncer server.
It still is missing a few things before we can complete this effort:
* A registration form for server admins to configure their server for
this service, with tests.
* Code (and tests) for actually bouncing the notifications.
This is mostly just moving methods out of compose.js.
The variable `is_composing_message`, which isn't a boolean, has
been renamed to `message_type`, and there are new functions
set_message_type() and get_message_type() that wrap it.
This commit removes some shims related to the global variable
`compose_state`; now, `compose_state` is a typical global
variable with a 1:1 relationship with the module by the same
name.
The new module has 100% line coverage, most of it coming
via the tests on compose_actions.js. (The methods here are
super simple, so it's a good thing that the tests are somewhat
integrated with a higher layer.)
This commit makes sure that GitHubAuthBackend will only authenticate
using its own authenticate method. This is done by adding a new
Python Social Auth strategy which instead of calling authenticate
method of Django, calls the authenticate of the backend directly.
The problem this commit solves is that while authenticating through
GitHub backend, we were ending up getting authenticated through
ZulipDummyBackend. This might happen because the default strategy used
by Python Social Auth calls the authenticate method of Django which
iterates over all the backends and tries the authenticate methods
which match with the function arguments. The new strategy this commit
adds calls the authenticate method of GitHub backend directly which
makes sense because we already know that we want to authenticate with
GithHub.
The actual problem of why we are ending up on ZulipDummyBackend is
still a mystery because the function arguments passed to its
authenticate method are different. It shouldn't be called.
We now wait to load Organization sections until you
click on the section (or virtually click by using arrow
keys).
Some of the sections are coupled in terms of their setup,
so some sections will already be loaded if you had clicked
on a related section.
This implements a list_render closure class that allows for
progressive, responsive rendering of long, scrollable lists, with
filtering support.
It isn't used, at present.
This module extracts these two functions that get called by
several other modules:
start()
cancel()
It is a little bit arbitrary which functions got pulled over
with them, but it's generally functions that would have only
been called via start/cancel.
There are two goals for splitting out this code. The first
goal is simply to make `compose.js` have fewer responsibilities.
The second goal is to help break up circular dependencies.
The extraction of this module does more to clarify
dependencies than actually break them. The methods start()
and cancel() had actually been shimmed in an earlier commit,
and now they no longer have a shim.
Besides start/cancel, most of the functions here are only
exported to facilitate test stubbing. An exception is
decorate_stream_bar(), which is currently called from
ui_init.js. We probably should move the "blur" handler out
of there, but cleaning up ui_init.js is a project for another
day.
It may seem slightly odd that this commit doesn't pull over
finish() into this module, but finish() would bring in the
whole send-message codepath. You can think of it like this:
* compose_actions basically just populates the compose box
* compose.finish() makes the compose box do its real job,
which is to send a message
Instead of zulip_test, use zulip_test_template for backend DB. This
makes sure that the DB used by backend tests is different from the
DB, which will be zulip_test, used by Casper tests.
This is mostly moving code, but we do add short-circuit logic
for some live-updating methods here.
Note that this affects two different sections of the admin app:
* Organization settings
* Authentication methods
We really want to move to one module per section, but there is some
legacy coupling that makes this difficult for now.
This fixes 2 issues:
* Some exceptions were not being properly emailed to admins.
* A bug in the parens placement in the default Zulip handlers list
resulted in the console/file handlers being accidentally excluded if
!ERROR_REPORTING.
Fixes#4127.
For the settings UI, we now wait until a user goes to a particular
settings section before calling the appropriate function to set
up the section (which usually involves setting up click handlers
and populating initial data).
The code here used to live in hotkey.js. Its complicated calling
protocol made it difficult to unit test. We are also trying to
slim down hotkey.js.
Our arrow navigation for things like `#stream_filters` has always
been kind of awkward, since it's difficult to get the focus to
their list items. This commit does nothing to fix that yet.
Django tries to authenticate against all backends one by one.
The authenticate() function of GitHub backend used to take
*args and **kwargs arguments due to which it could be called
against any set of arguments. Django uses arguments to
differentiate authenticate() methods.
Most of this code was simply moved from activity.js with some
minor renaming of functions like set_presence_info -> set_info.
Some functions were slightly nontrivial extractions:
is_not_offline:
came from activity.huddle_fraction_present
get_status/get_mobile:
simple getters
set_user_status:
partial extraction from activity.set_user_status
last_active_date:
pulled out of admin.js code
We also fixed activity.filter_and_sort to take user_ids.
This moves the implementations of error/report/message from
ui.js to ui_report.js. They had been shimmed before, so calling
modules still use the same names to call the functions, but we
no longer need the shims.
This commit adds the backend support for a new style of tutorial which
allows for highlighting of multiple areas of the page with hotspots that
disappear when clicked by the user.
This adds helpful email notifications for users who just logged into a
Zulip server, as a security protection against accounts being hacked.
Text tweaked by tabbott.
Fixes#2182.
This change moves most of the logic related to starting and
stopping outbound typing indicators to a new module called
typing_status.js that is heavily unit tested.
While this was in some sense a rewrite, the logic was mostly
inspired by the existing code.
This change does fix one known bug, which is that when we
were changing recipients before (while typing was active), we
were not stopping and starting typing indicators. This was
a fairly minor bug, since usually users leave the compose
box to change recipients, and we would do stop/start under
that scenario. Now we also handle the case where the user
does not leave the compose box to change recipients.
Send typing notification events when user types in the compose box.
Listen for these events and display a notification.
Sending notifications: Notifications are throttled, so that start
notifications are sent every 10 seconds of active typing, and stop
notifications are sent 5 seconds after active typing stops or when the
compose box is closed.
Displaying notifications:
When a typing notification is received, if the current narrow is private
messages or is: pm-with and the user is not the sender,
"Othello is typing..." is displayed underneath the last message. This notification is
removed after 15 seconds. If another notification is received during this period, the
expiration is extended. When a stop notification is received the notification is removed.
Internally, a list of users currently typing is maintained for each
conversation (in a dict). When an event is received the list (for the appropriate
conversation) is updated and the notifications template is re-rendered
based on the narrow information. This template is also re-rendered when
the narrow changes.
Significantly modified by tabbott for clarity.
Fixes#150.
This fixes an issue where if you saved a Python file (even just
changing whitespace) while casper tests were running, the Tornado
server being used would restart, triggering a confusing error like
this:
ReferenceError: Can't find variable: $
Traceback:
undefined:2
:4
Suite explicitly interrupted without any message given.
Django 1.10 has changed the implementation of this function to
match our custom implementation; in addition to this, we prefer
render().
Fixes#1914 via #4093.
This consolidates all actions to close modals into modals.js and
triggers the correct cleaning/collapsing function dependent on what the
data-overlay attribute is labeled as.
It also ensures these all have an e.stopPropagation().
Fixes#4029.
This fixes the mobile web experience for Chrome on iOS.
Apparently, Chrome-on-iOS silently has a `viewport` module that
overrides and user-defined module by that name, causing all of our
code that accesses the viewport module to not work on that platform.
We fix this by renaming it.
- Add settings parameter for max realm icon size.
- Add settings parameter for max user avatar size.
- Add checking file size to avatar and icon
uploading views.
- Transfer file size limit parameter to frontend.
- Add tests.
- Add `OFFLINE_THRESHOLD_SECS` settings parameter
to handle offline period.
- Set aggregated status to offline if user's status
haven't changed for `OFFLINE_THRESHOLD_SECS` period.
- Add test for offline aggregated status.
This feature hardcoded zulip.com, and never really made much sense
("feedback" should generally go to the local server administrator, not
to the Zulip development community).
This module handles the popovers in the stream list--one for
stream actions and another for topic-specific actions.
The extraction was mostly straightforward, but I did move some
of the code related to the color picker to be more consistent
with how I organized the other click handlers.
- Remove `handlebars.runtime.js` from static/third and fetch it from npm
- Upgrade `handlebars` to 3.0.3.
I change the test since there is a patch about line, written in
handlebars'
v2.0.0-beta.1 release note:
"Lines containing only block statements and whitespace are now removed."
Fixes part of #1709.
Eeshan Garg
Tim Abbott
Juliana Bacelar
Vishnu Ks
julianasobreira
Supermanu
Robert Hönig
Brock Whittaker
Steve Howell
Umair Khan
Vaida Plankyte
Aditya Bansal
Greg Price
rht
Jack Zhang
neiljp (Neil Pilgrim)
Rishi Gupta
James Rowan
Steve Howell
vbNETonIce
Pweaver (Paul Weaver)
Cory Lynch
Harshit Bansal
Mehanig
Rafid Aslam
Akhil
K.Kanakhin
Rohitt Vashishtha
umkay
Eklavya Sharma
kunall17
Mahim Goyal
Jack Zhang
Shayan Toqraee
Amy Liu
hollywoodno
sinwar
Raghav Jajodia
Yago González
wangjames
Arpith Siromoney