Hello everyone! I have a question about uploading subject set.
In my project we have to upload from 500 to 1000 images to be classified. I have read in the instructions that it is possible to not write one row at a time in the manifest but there is a easier and faster way but I dont' understand what I have to do.
There is someone that can help me?
Hello.
Based on my past troubles, I suggest checking that your filenames are all correctly spelled in your manifest. If so, check the image that wasn't uploaded to see if there's any issue with it - check that it opens fine in an image viewer. Also be sure no images are larger than the size limit. I believe that's still 1MB.
I hope you get it figured out!
-Marianne (Monkey Health Explorer)
Hello. I'm new here and am trying to figure out how to set up a project. I initially struggled to get a manifest and images to load on the project page. I managed to accomplish that but while I had five entries on the manifest the uploader only recognized four images. I don't understand what happened because all five lines had the same type of information and everything was loaded at the same time. Anybody have an idea what happened? Thank you for your help.
I hope to expand it to a larger public project in the future where time constraints aren't an issue. As I said, this project is basically a internal project and a beta to test the effectiveness of the citzen science approach. I'll most certainly employ known subjects into training in the future!
Re weighting - You can not "fix" poor or insufficient data by any weighting scheme.
The first thing I would suggest is that "time constraints" that result in the choice of low retirement limits is a very fast way of wasting everyone's time, and the first go-to solution for improving results (or at least knowing the accuracy of those results) is increased classifications counts . Certainly avoid the lure of weighting as a band aid for poor or insufficient data.
How do you measure "experience" or compare a volunteer to the consensus if they only do a few classifications? In a typical project something like 80% of the classifications are done by volunteers that do 5 or less classifications in total! ( FossilFinders)
Weighting has some value if you force volunteers to classify training sets or have a high percent of known subjects seeded into their early efforts AND you can retain those volunteers after training.
It's only really valuable for a PUBLIC project where someone malicious can put random classifications.
I recommend checking out Willet+2013 that Hayley linked earlier as that's where I started. You should be able to put together code that uses your voting fractions to calculate a consistency value for each question and then average it at the end.
Thanks for sharing this. I'm quite new to this kind of research but I found your post very insightful. The idea of combining experience with consistency in weighting sounds quite interesting, especially for improving early-stage projects like yours.
Do you think this method might also help reduce noise from occasional random classifications?
Looking forward to seeing how your project develops!
I understand a lot of what you are saying. I also wish I weren't limited by time and could increase the retirement limit. Hopefully, by the time I have a PhD, I'll be able to do a project of the scale to bring back the Crowd Wisdom you talk about.
As I said in my response to Hailey, I'll likely forgo it for now and use this dataset as a potential 'gold standard' subject list for the future, more extensive project.
I wish I could increase the classification limit, but unfortunately, as I mentioned before, I graduate from my university this semester and have limited time.
The plan for this project was to (1) expand our current catalogues for these subjects, (2) do a beta test on the citizen science approach to classifying the objects I provided, and (3) improve the classification scheme for a future, more extensive project. With the upcoming surveys, we expect our subject list to grow in number, and the citizen science approach excels at dealing with large dataset classifications. I guess I would ultimately consider this project the "training" sample.
I appreciate the additional weighting schemes you suggested, but you are right that they add more complexity. I think I'll forgo them for this project. Depending on the final number, I'll look into either unretiring the tied subjects and getting some additional classifications or going through them individually with my group.
I would love to bring this to the more considerable Galaxy Zoo contribution as well, but that would likely be a while from now. My mentor may be interested in the future as the upcoming datasets from the allsky surveys are extremely promising for our niche projects!
My experiences with weighting, over several different projects is that the benefits if any, depend on the project task difficulty/uncertainty and the nature of the pool of volunteers the project attracts. I have seen no significant advantage with various weighting schemes across several projects where the volunteer pool was "open" - ie with no experience or training filtering.
Projects that attract many volunteers but with low retention rates (low classifications per volunteer) are a particular issue - unless one has a high percentage of gold standard subjects, or forces volunteers to classify a known "training set" there is simply too few volunteers with sufficient classifications on known subjects (or "high consensus " subjects) to generate a worthwhile weighting.
The higher the uncertainty for a subject classification, the weaker consensus becomes and hence the more problematic it becomes to weight volunteers against "consensus". This is a particular problem if you are counting on weighting to improve consensus - one can easily devise a weighting scheme that has a strong confirmation bias and actually generates false consensus.
As I look around zooniverse today, looking only at live, non-transcription projects, I find many projects that are far removed from the early work in citizen science that discussed Crowd Wisdom. I see many projects with "early retirement" rules (many set at as low as 3 classifications) or customized retirement pipelines. I see project with retirement limits of as low as 2, a median of about 10 over the current active projects and very few projects with 30 or 40 classifications - 10 does not make a crowd. This link to current active workflow stats shows the range of retirement limits currently in use. The first thing I would suggest is that "time constraints" that result in the choice of low retirement limits is a very fast way of wasting everyone's time, and the first go-to solution for improving results (or at least knowing the accuracy of those results) is increased classifications counts . Certainly avoid the lure of weighting as a band aid for poor or insufficient data.
Thanks for your question! For some context, I'm a member of the Zooniverse team and Galaxy Zoo collaboration, so I'll try to answer from both perspectives.
Broadly, most projects see some slight benefit from implementing user weighting, but often not enough to warrant the added complexity. Notably, to do user weighting well, it's helpful to have lots of classifications per user or a set of training subjects that you can use to assess user accuracy.
The weighting scheme you mentioned from Willett+2013 has a pretty small impact on actual Galaxy Zoo data. This weighting is only applied to about ~5% of users, and the mean change in vote fraction across galaxies is <0.003 (Section 4.3.1 in Walmsley+2022). For a project with a small retirement limit, changes this small will have little to no impact on your data.
We did use a stronger weighting scheme in Galaxy Zoo: Weird & Wonderful, our anomaly identification project. However, our method relied on upweighting experienced Galaxy Zoo users who also classified on that project. You can read a bit about that in Mantha+2024.
Outside of Galaxy Zoo, we've also seen different weighting schemes implemented across other projects, such as Space Warps (Marshall+2016) and, as you mentioned, Active Asteroids (Chandler+2024). Both of these examples require "training" subjects (known subjects with known classifications) that are used to train volunteers, assess their ability, or calculate their weighting. If this is something you would like to pursue, you can reach out to us at contact@zooniverse.org to get this set up. However, you mentioned you're on a time constraint, so retraining your users on these subjects may not be feasible.
I imagine all these options are over-engineered solutions to something that may just be solved with a higher retirement limit. Requiring an even number of classifications is fine — your project has 5 classification options, so there's always a chance of a tied plurality, no matter the number of classifications before retirement. Galaxy Zoo, for example, requires 40 classifications. My suggestion would be to increase your retirement limit and unretire all subjects (or even just those that have low consensus fractions). Here's an example of how you can do that with a Python script: Retiring and Unretiring Subjects.
If you have any interest in bringing this to the Galaxy Zoo collaboration, feel free to message me and we can find time to chat.
I hope something in there is helpful! Do let me know if there's any other information I can provide.
I'm currently working on a small-scale Galaxy Zoo project that will essentially serve as a beta for a future more big project. I'm currently testing it with my research group and some other volunteers and everything is going well.
With some of the initial results, I've calculated the consistency per user using the method outlined by Willett+2013 and have also attempted their weighting function. However, I'm not sure how effective their weighting function will be with my results as, so far, most users lie around 0.8-0.9 in consistency values. Does anyone have any advice here?
I've also considered implementing an 'experience' based weighting function that includes consistency too as my beta is still a bit difficult for newcomers. Does anyone know the effectiveness of these types of weighting functions? I've found a recent paper (Chandler+2024) that suggests that there is some merit to it, but I thought I would ask.
Other than that, I've had some issues with tied classifications as I'm currently using a low and even subject retirement count due to time constraints. I suspect this will be less of the case with a larger scale project and choosing an odd retirement count. I also suspect an 'experience' based weighting function may help here, but I'm unsure.
As far as I know you can create your own. The template ones are .svg images but I believe .png images will work as well. The trick is to draw them so they look good when reduced to the small icon size.
The project Cedar Creek has .png filtericons which they created. You can copy and borrow from any project (it would be polite to ask of course but I would be astounded if anyone ever was less than happy to share!)
There have been projects in the past that had a server somewhere so that volunteers could upload their images, which were then used as zooniverse subjects, but the site to upload images was totally separate from zooniverse.
Volunteers can upload images to several third party hosting sites and then link to them in the comments section - all that is needed is that the image have a url that displays as an image directly in a browser ie NOT viewed through some viewer maintained by the host - example google or dropbox images can only be viewed through that site's viewer, they do not have a standalone url.
Many of these sites have a "free" version which is add supported, or have limited uploads, short expiry times or at some point they change the rules and your images are lost . - example: FossilFinders had a volunteer that put in many hundreds of hours collecting and stitching together zooniverse subjects in panoramic views of a whole area... their images were hosted on some site that changed the rules and became a pay site - all those hours lost and hundreds of broken links in FossilFinder!
Hi @PC_Outreach_MNS and good morning to you as well. You can make changes while the project is marked as finished. The point at which we'll need to update the project status is when you're getting ready to re-launch, so just be sure to send us an email before that point and we'll talk through your options. For example, if you're making significant changes to the workflows you may want to run a new beta test, but if they're exactly the same as before we'll just discuss the re-launch process, comms, set a date, etc.
Page of 210
Talk is a place for Zooniverse volunteers and researchers to discuss their projects, collect and share data, and work together to make new discoveries.
May 4th 2025, 6:25 am
Theories about the existence of white holes are being sought.
April 17th 2025, 8:39 am
Hello everyone! I have a question about uploading subject set.
In my project we have to upload from 500 to 1000 images to be classified. I have read in the instructions that it is possible to not write one row at a time in the manifest but there is a easier and faster way but I dont' understand what I have to do.
There is someone that can help me?
Thank you very much!
April 10th 2025, 2:30 pm
Hello.
Based on my past troubles, I suggest checking that your filenames are all correctly spelled in your manifest. If so, check the image that wasn't uploaded to see if there's any issue with it - check that it opens fine in an image viewer. Also be sure no images are larger than the size limit. I believe that's still 1MB.
I hope you get it figured out!
-Marianne (Monkey Health Explorer)
April 10th 2025, 1:21 pm
Hello. I'm new here and am trying to figure out how to set up a project. I initially struggled to get a manifest and images to load on the project page. I managed to accomplish that but while I had five entries on the manifest the uploader only recognized four images. I don't understand what happened because all five lines had the same type of information and everything was loaded at the same time. Anybody have an idea what happened? Thank you for your help.
April 10th 2025, 6:16 am
Yes, I understand!
I hope to expand it to a larger public project in the future where time constraints aren't an issue. As I said, this project is basically a internal project and a beta to test the effectiveness of the citzen science approach. I'll most certainly employ known subjects into training in the future!
-Trevor
April 9th 2025, 11:19 pm
Re weighting - You can not "fix" poor or insufficient data by any weighting scheme.
The first thing I would suggest is that "time constraints" that result in the choice of low retirement limits is a very fast way of wasting everyone's time, and the first go-to solution for improving results (or at least knowing the accuracy of those results) is increased classifications counts . Certainly avoid the lure of weighting as a band aid for poor or insufficient data.
How do you measure "experience" or compare a volunteer to the consensus if they only do a few classifications? In a typical project something like 80% of the classifications are done by volunteers that do 5 or less classifications in total! ( FossilFinders)
Weighting has some value if you force volunteers to classify training sets or have a high percent of known subjects seeded into their early efforts AND you can retain those volunteers after training.
April 9th 2025, 8:54 pm
Hi Ali,
Absolutely!
It's only really valuable for a PUBLIC project where someone malicious can put random classifications.
I recommend checking out Willet+2013 that Hayley linked earlier as that's where I started. You should be able to put together code that uses your voting fractions to calculate a consistency value for each question and then average it at the end.
-Trevor
April 9th 2025, 8:43 pm
Hi Trevor,
Thanks for sharing this. I'm quite new to this kind of research but I found your post very insightful. The idea of combining experience with consistency in weighting sounds quite interesting, especially for improving early-stage projects like yours.
Do you think this method might also help reduce noise from occasional random classifications?
Looking forward to seeing how your project develops!
— Alimualia_01
April 3rd 2025, 3:34 am
Hello Pmason,
Thank you for your comment!
I understand a lot of what you are saying. I also wish I weren't limited by time and could increase the retirement limit. Hopefully, by the time I have a PhD, I'll be able to do a project of the scale to bring back the Crowd Wisdom you talk about.
As I said in my response to Hailey, I'll likely forgo it for now and use this dataset as a potential 'gold standard' subject list for the future, more extensive project.
Trevor
April 3rd 2025, 3:21 am
Hi Hayley,
Thank you for the in-depth response!
I wish I could increase the classification limit, but unfortunately, as I mentioned before, I graduate from my university this semester and have limited time.
The plan for this project was to (1) expand our current catalogues for these subjects, (2) do a beta test on the citizen science approach to classifying the objects I provided, and (3) improve the classification scheme for a future, more extensive project. With the upcoming surveys, we expect our subject list to grow in number, and the citizen science approach excels at dealing with large dataset classifications. I guess I would ultimately consider this project the "training" sample.
I appreciate the additional weighting schemes you suggested, but you are right that they add more complexity. I think I'll forgo them for this project. Depending on the final number, I'll look into either unretiring the tied subjects and getting some additional classifications or going through them individually with my group.
I would love to bring this to the more considerable Galaxy Zoo contribution as well, but that would likely be a while from now. My mentor may be interested in the future as the upcoming datasets from the allsky surveys are extremely promising for our niche projects!
Trevor
April 2nd 2025, 6:21 pm
My experiences with weighting, over several different projects is that the benefits if any, depend on the project task difficulty/uncertainty and the nature of the pool of volunteers the project attracts. I have seen no significant advantage with various weighting schemes across several projects where the volunteer pool was "open" - ie with no experience or training filtering.
Projects that attract many volunteers but with low retention rates (low classifications per volunteer) are a particular issue - unless one has a high percentage of gold standard subjects, or forces volunteers to classify a known "training set" there is simply too few volunteers with sufficient classifications on known subjects (or "high consensus " subjects) to generate a worthwhile weighting.
The higher the uncertainty for a subject classification, the weaker consensus becomes and hence the more problematic it becomes to weight volunteers against "consensus". This is a particular problem if you are counting on weighting to improve consensus - one can easily devise a weighting scheme that has a strong confirmation bias and actually generates false consensus.
As I look around zooniverse today, looking only at live, non-transcription projects, I find many projects that are far removed from the early work in citizen science that discussed Crowd Wisdom. I see many projects with "early retirement" rules (many set at as low as 3 classifications) or customized retirement pipelines. I see project with retirement limits of as low as 2, a median of about 10 over the current active projects and very few projects with 30 or 40 classifications - 10 does not make a crowd. This link to current active workflow stats shows the range of retirement limits currently in use. The first thing I would suggest is that "time constraints" that result in the choice of low retirement limits is a very fast way of wasting everyone's time, and the first go-to solution for improving results (or at least knowing the accuracy of those results) is increased classifications counts . Certainly avoid the lure of weighting as a band aid for poor or insufficient data.
April 2nd 2025, 4:06 pm
Hi Trevor,
Thanks for your question! For some context, I'm a member of the Zooniverse team and Galaxy Zoo collaboration, so I'll try to answer from both perspectives.
Broadly, most projects see some slight benefit from implementing user weighting, but often not enough to warrant the added complexity. Notably, to do user weighting well, it's helpful to have lots of classifications per user or a set of training subjects that you can use to assess user accuracy.
The weighting scheme you mentioned from Willett+2013 has a pretty small impact on actual Galaxy Zoo data. This weighting is only applied to about ~5% of users, and the mean change in vote fraction across galaxies is <0.003 (Section 4.3.1 in Walmsley+2022). For a project with a small retirement limit, changes this small will have little to no impact on your data.
We did use a stronger weighting scheme in Galaxy Zoo: Weird & Wonderful, our anomaly identification project. However, our method relied on upweighting experienced Galaxy Zoo users who also classified on that project. You can read a bit about that in Mantha+2024.
Outside of Galaxy Zoo, we've also seen different weighting schemes implemented across other projects, such as Space Warps (Marshall+2016) and, as you mentioned, Active Asteroids (Chandler+2024). Both of these examples require "training" subjects (known subjects with known classifications) that are used to train volunteers, assess their ability, or calculate their weighting. If this is something you would like to pursue, you can reach out to us at contact@zooniverse.org to get this set up. However, you mentioned you're on a time constraint, so retraining your users on these subjects may not be feasible.
I imagine all these options are over-engineered solutions to something that may just be solved with a higher retirement limit. Requiring an even number of classifications is fine — your project has 5 classification options, so there's always a chance of a tied plurality, no matter the number of classifications before retirement. Galaxy Zoo, for example, requires 40 classifications. My suggestion would be to increase your retirement limit and unretire all subjects (or even just those that have low consensus fractions). Here's an example of how you can do that with a Python script: Retiring and Unretiring Subjects.
If you have any interest in bringing this to the Galaxy Zoo collaboration, feel free to message me and we can find time to chat.
I hope something in there is helpful! Do let me know if there's any other information I can provide.
Hayley
March 29th 2025, 12:01 am
Hi all!
I'm currently working on a small-scale Galaxy Zoo project that will essentially serve as a beta for a future more big project. I'm currently testing it with my research group and some other volunteers and everything is going well.
With some of the initial results, I've calculated the consistency per user using the method outlined by Willett+2013 and have also attempted their weighting function. However, I'm not sure how effective their weighting function will be with my results as, so far, most users lie around 0.8-0.9 in consistency values. Does anyone have any advice here?
I've also considered implementing an 'experience' based weighting function that includes consistency too as my beta is still a bit difficult for newcomers. Does anyone know the effectiveness of these types of weighting functions? I've found a recent paper (Chandler+2024) that suggests that there is some merit to it, but I thought I would ask.
Other than that, I've had some issues with tied classifications as I'm currently using a low and even subject retirement count due to time constraints. I suspect this will be less of the case with a larger scale project and choosing an odd retirement count. I also suspect an 'experience' based weighting function may help here, but I'm unsure.
Thank you for your time!
-Trevor
March 28th 2025, 1:56 pm
I also want to do this project
March 21st 2025, 9:30 am
Thank you so much @Pmason , I will evaluate the best solution for our project. I didn't know CitSci.org platform.
March 20th 2025, 4:24 pm
This discussion may be of significance to your situation:
March 20th 2025, 8:07 am
Okay, thank you very much for your answer!
March 18th 2025, 8:29 pm
As far as I know you can create your own. The template ones are .svg images but I believe .png images will work as well. The trick is to draw them so they look good when reduced to the small icon size.
The project Cedar Creek has .png filtericons which they created. You can copy and borrow from any project (it would be polite to ask of course but I would be astounded if anyone ever was less than happy to share!)
The Colorado Corridors project had some very nice "homemade" graphics in their field guide which I know were shared around.
https://www.zooniverse.org/projects/coloradocorridorsproject/colorado-corridors-project/talk
Unfortunately we can not currently see their filtericons
The Mont Blanc filtericons are also .png and I believe "homemade". I think any small high contrast .png file ( example 75X75 pixels) is suitable.
March 18th 2025, 7:52 pm
There have been projects in the past that had a server somewhere so that volunteers could upload their images, which were then used as zooniverse subjects, but the site to upload images was totally separate from zooniverse.
Volunteers can upload images to several third party hosting sites and then link to them in the comments section - all that is needed is that the image have a url that displays as an image directly in a browser ie NOT viewed through some viewer maintained by the host - example google or dropbox images can only be viewed through that site's viewer, they do not have a standalone url.
Many of these sites have a "free" version which is add supported, or have limited uploads, short expiry times or at some point they change the rules and your images are lost . - example: FossilFinders had a volunteer that put in many hundreds of hours collecting and stitching together zooniverse subjects in panoramic views of a whole area... their images were hosted on some site that changed the rules and became a pay site - all those hours lost and hundreds of broken links in FossilFinder!
March 18th 2025, 4:54 pm
Hi @PC_Outreach_MNS and good morning to you as well. You can make changes while the project is marked as finished. The point at which we'll need to update the project status is when you're getting ready to re-launch, so just be sure to send us an email before that point and we'll talk through your options. For example, if you're making significant changes to the workflows you may want to run a new beta test, but if they're exactly the same as before we'll just discuss the re-launch process, comms, set a date, etc.